2024-04-29T20:04:17.702 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-29T20:04:17.702 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_lflores@teuthology --verbose --archive /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692 --name lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/orig.config.yaml 2024-04-29T20:04:17.742 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-29T20:04:17.846 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692 branch: wip-yuri6-testing-2024-04-02-1310 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7678692' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-04-02-1310 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 osd op thread timeout: 150 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 354447ca5357e926795d009be846687f04556beb valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 354447ca5357e926795d009be846687f04556beb selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2024-04-02-1310 sha1: 354447ca5357e926795d009be846687f04556beb owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1434 sha1: 354447ca5357e926795d009be846687f04556beb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2024-04-02-1310 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 354447ca5357e926795d009be846687f04556beb targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNd2mtGOrCmynAlbVYW38ivB5bNl5QYQUD0ObRY1s05WIO5wjEpeyExsQSztEFoky8ntBJ1vZjctQHDJnc1VKaE= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKi49WANi6PnJX4+APe3vVQt93GlNunHK+tM60aoulD8adVf1+KW+zXdQtOTHwjo77GVRci9+1MwUWCFqAxtVwk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-29_19:49:34 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-29T20:04:17.847 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa; will attempt to use it 2024-04-29T20:04:17.847 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks 2024-04-29T20:04:17.847 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-29T20:04:17.849 INFO:teuthology.task.internal:Checking packages... 2024-04-29T20:04:17.873 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '354447ca5357e926795d009be846687f04556beb' 2024-04-29T20:04:17.873 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T20:04:17.873 INFO:teuthology.packaging:ref: None 2024-04-29T20:04:17.873 INFO:teuthology.packaging:tag: None 2024-04-29T20:04:17.873 INFO:teuthology.packaging:branch: wip-yuri6-testing-2024-04-02-1310 2024-04-29T20:04:17.873 INFO:teuthology.packaging:sha1: 354447ca5357e926795d009be846687f04556beb 2024-04-29T20:04:17.873 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 2024-04-29T20:04:18.012 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3150.g354447ca 2024-04-29T20:04:18.013 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-29T20:04:18.022 INFO:teuthology.task.internal:no buildpackages task found 2024-04-29T20:04:18.022 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-29T20:04:18.045 INFO:teuthology.task.internal:Saving configuration 2024-04-29T20:04:18.061 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-29T20:04:18.069 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T20:04:18.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 19:58:28.948395', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNd2mtGOrCmynAlbVYW38ivB5bNl5QYQUD0ObRY1s05WIO5wjEpeyExsQSztEFoky8ntBJ1vZjctQHDJnc1VKaE='} 2024-04-29T20:04:18.113 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 19:58:28.950488', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKi49WANi6PnJX4+APe3vVQt93GlNunHK+tM60aoulD8adVf1+KW+zXdQtOTHwjo77GVRci9+1MwUWCFqAxtVwk='} 2024-04-29T20:04:18.113 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-29T20:04:18.121 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-29T20:04:18.121 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-29T20:04:18.121 INFO:teuthology.run_tasks:Running task console_log... 2024-04-29T20:04:18.187 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcb8e717f70>, signals=[15]) 2024-04-29T20:04:18.188 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-29T20:04:18.196 INFO:teuthology.task.internal:Opening connections... 2024-04-29T20:04:18.196 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:04:18.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T20:04:18.277 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:04:18.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T20:04:18.353 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-29T20:04:18.359 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-04-29T20:04:18.378 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-04-29T20:04:18.379 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-29T20:04:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-04-29T20:04:18.438 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-04-29T20:04:18.438 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-29T20:04:18.438 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-04-29T20:04:18.438 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-29T20:04:18.438 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-29T20:04:18.438 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-29T20:04:18.438 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-04-29T20:04:18.461 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-04-29T20:04:18.479 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-04-29T20:04:18.480 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-04-29T20:04:18.537 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2024-04-29T20:04:18.538 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-29T20:04:18.538 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-04-29T20:04:18.538 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-29T20:04:18.538 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-29T20:04:18.538 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-29T20:04:18.538 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-04-29T20:04:18.552 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-29T20:04:18.567 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-29T20:04:18.575 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-29T20:04:18.575 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-04-29T20:04:18.577 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-04-29T20:04:18.594 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-29T20:04:18.600 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-29T20:04:18.601 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-04-29T20:04:18.635 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-04-29T20:04:18.674 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-29T20:04:18.726 INFO:teuthology.run_tasks:Running task kernel... 2024-04-29T20:04:18.742 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-29T20:04:18.742 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-29T20:04:18.742 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-29T20:04:18.743 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-29T20:04:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:04:18.760 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-29T20:04:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:04:18.761 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-04-29T20:04:18.817 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-437.el9.x86_64 2024-04-29T20:04:18.817 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-437.el9.x86_64 2024-04-29T20:04:18.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-04-29T20:04:18.820 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-437.el9.x86_64 2024-04-29T20:04:18.820 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-437.el9.x86_64 2024-04-29T20:04:18.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-04-29T20:04:21.257 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-04-29T20:04:24.966 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-04-29T20:04:26.151 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-04-29T20:04:27.475 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-29T20:04:32.405 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.2 MB 00:01 2024-04-29T20:04:34.239 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-29T20:04:35.909 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-29T20:04:37.951 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 781 kB/s | 6.2 MB 00:08 2024-04-29T20:04:39.582 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-29T20:04:40.611 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-29T20:04:40.907 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-29T20:04:42.763 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-29T20:04:42.763 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-29T20:04:42.849 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:04:42.850 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:04:42.850 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:04:42.986 DEBUG:teuthology.orchestra.run.smithi167:> echo no | sudo yum reinstall kernel || true 2024-04-29T20:04:43.485 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:03 ago on Mon 29 Apr 2024 08:04:40 PM UTC. 2024-04-29T20:04:43.618 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:04:43.628 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:04:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout:Reinstalling: 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 255 k 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 0 2024-04-29T20:04:43.629 INFO:teuthology.orchestra.run.smithi167.stderr:Operation aborted. 2024-04-29T20:04:43.690 INFO:teuthology.orchestra.run.smithi167.stdout:Is this ok [y/N]: 2024-04-29T20:04:43.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum reinstall -y kernel || true 2024-04-29T20:04:44.184 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:04 ago on Mon 29 Apr 2024 08:04:40 PM UTC. 2024-04-29T20:04:44.300 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout:Reinstalling: 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:04:44.301 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:04:44.302 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 255 k 2024-04-29T20:04:44.302 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 0 2024-04-29T20:04:44.302 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:04:44.526 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-29T20:04:44.526 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:04:44.526 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.1 MB/s | 255 kB 00:00 2024-04-29T20:04:44.541 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:04:44.752 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:04:44.753 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:04:44.758 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:04:44.758 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:04:44.887 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:04:44.977 INFO:teuthology.orchestra.run.smithi167.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-29T20:04:44.977 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-29T20:04:45.696 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-29T20:04:45.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-29T20:04:45.858 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-29T20:04:46.407 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-29T20:04:46.407 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:04:46.408 INFO:teuthology.orchestra.run.smithi167.stdout:Reinstalled: 2024-04-29T20:04:46.408 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-29T20:04:46.408 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:04:46.408 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:04:46.597 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T20:04:46.644 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-29T20:04:46.644 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-29T20:04:46.644 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-29T20:04:46.645 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-29T20:04:46.645 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-29T20:04:46.674 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-04-29T20:04:46.696 DEBUG:teuthology.parallel:result is None 2024-04-29T20:04:47.865 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-29T20:04:47.865 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-29T20:04:47.960 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:04:47.961 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:04:47.962 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:04:48.080 DEBUG:teuthology.orchestra.run.smithi064:> echo no | sudo yum reinstall kernel || true 2024-04-29T20:04:48.605 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:03 ago on Mon 29 Apr 2024 08:04:45 PM UTC. 2024-04-29T20:04:48.747 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:04:48.760 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:04:48.760 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-04-29T20:04:48.760 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:04:48.760 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-04-29T20:04:48.760 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 255 k 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-04-29T20:04:48.761 INFO:teuthology.orchestra.run.smithi064.stderr:Operation aborted. 2024-04-29T20:04:48.816 INFO:teuthology.orchestra.run.smithi064.stdout:Is this ok [y/N]: 2024-04-29T20:04:48.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum reinstall -y kernel || true 2024-04-29T20:04:49.335 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:04 ago on Mon 29 Apr 2024 08:04:45 PM UTC. 2024-04-29T20:04:49.456 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:04:49.456 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 255 k 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-04-29T20:04:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:04:49.752 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-04-29T20:04:49.752 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:04:49.752 INFO:teuthology.orchestra.run.smithi064.stdout:Total 864 kB/s | 255 kB 00:00 2024-04-29T20:04:49.767 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:04:49.953 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:04:49.953 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:04:49.960 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:04:49.960 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:04:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:04:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-29T20:04:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-29T20:04:50.818 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-29T20:04:50.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-29T20:04:51.493 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-29T20:04:51.493 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:04:51.493 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalled: 2024-04-29T20:04:51.493 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-29T20:04:51.493 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:04:51.493 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:04:51.675 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T20:04:51.723 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-29T20:04:51.723 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-29T20:04:51.723 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-29T20:04:51.724 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-29T20:04:51.724 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-29T20:04:51.760 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-04-29T20:04:51.776 DEBUG:teuthology.parallel:result is None 2024-04-29T20:04:51.777 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-29T20:04:51.784 INFO:teuthology.task.internal:Creating test directory... 2024-04-29T20:04:51.784 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-29T20:04:51.819 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-29T20:04:51.836 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-29T20:04:51.848 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-29T20:04:51.862 INFO:teuthology.task.internal:Creating archive directory... 2024-04-29T20:04:51.862 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-29T20:04:51.873 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-29T20:04:51.905 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-29T20:04:51.916 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-29T20:04:51.916 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-29T20:04:51.937 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-29T20:04:51.968 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T20:04:51.970 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T20:04:51.978 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T20:04:51.982 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T20:04:51.983 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-29T20:04:51.994 INFO:teuthology.task.internal:Configuring sudo... 2024-04-29T20:04:51.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-29T20:04:52.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-29T20:04:52.049 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-29T20:04:52.062 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-29T20:04:52.062 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-29T20:04:52.087 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-29T20:04:52.106 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T20:04:52.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T20:04:52.227 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T20:04:52.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T20:04:52.348 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:04:52.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-29T20:04:52.412 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T20:04:52.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T20:04:52.507 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T20:04:52.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T20:04:52.634 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:04:52.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-29T20:04:52.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-04-29T20:04:52.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-04-29T20:04:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T20:04:52.788 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T20:04:53.203 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-29T20:04:53.211 INFO:teuthology.task.internal:Starting timer... 2024-04-29T20:04:53.211 INFO:teuthology.run_tasks:Running task pcp... 2024-04-29T20:04:53.225 INFO:teuthology.run_tasks:Running task selinux... 2024-04-29T20:04:53.235 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-29T20:04:53.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-04-29T20:04:53.273 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-04-29T20:04:53.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-04-29T20:04:53.340 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-04-29T20:04:53.343 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-29T20:04:53.343 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-04-29T20:04:53.361 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-04-29T20:04:53.362 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-04-29T20:04:53.392 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-04-29T20:04:53.392 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-04-29T20:04:53.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-29T20:04:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:04:53.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-29T20:04:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:04:53.458 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-29T20:04:53.458 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-04-29T20:04:53.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-04-29T20:04:53.522 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-29T20:04:53.531 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-29T20:04:53.550 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-29T20:04:53.552 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-29T20:08:25.145 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-04-29T20:08:25.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-29T20:08:25.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T20:08:25.222 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-04-29T20:08:25.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-29T20:08:25.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-29T20:08:25.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T20:08:25.382 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-04-29T20:08:25.464 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-29T20:08:25.465 INFO:teuthology.run_tasks:Running task clock... 2024-04-29T20:08:25.477 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-29T20:08:25.477 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-29T20:08:25.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T20:08:25.479 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-29T20:08:25.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T20:08:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-29T20:08:25.523 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-29T20:08:25.532 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-29T20:08:25.546 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-29T20:08:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-04-29T20:08:25.561 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-04-29T20:08:25.578 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-29T20:08:25.578 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2024-04-29T20:08:25.589 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2024-04-29T20:08:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-29T20:08:25.602 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-29T20:08:25.615 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-29T20:08:25.648 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-04-29T20:08:25.651 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T20:08:25.652 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-04-29T20:08:25.652 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.652 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.652 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.652 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.669 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-04-29T20:08:25.672 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T20:08:25.672 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-04-29T20:08:25.672 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.673 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.673 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.673 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:08:25.673 INFO:teuthology.run_tasks:Running task install... 2024-04-29T20:08:25.681 DEBUG:teuthology.task.install:project ceph 2024-04-29T20:08:25.681 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '354447ca5357e926795d009be846687f04556beb'}} 2024-04-29T20:08:25.681 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '354447ca5357e926795d009be846687f04556beb'} 2024-04-29T20:08:25.681 INFO:teuthology.task.install:Using flavor: default 2024-04-29T20:08:25.687 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-04-29T20:08:25.687 INFO:teuthology.task.install:extra packages: [] 2024-04-29T20:08:25.688 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '354447ca5357e926795d009be846687f04556beb', 'tag': None, 'wait_for_package': False} 2024-04-29T20:08:25.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=354447ca5357e926795d009be846687f04556beb 2024-04-29T20:08:25.690 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '354447ca5357e926795d009be846687f04556beb', 'tag': None, 'wait_for_package': False} 2024-04-29T20:08:25.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=354447ca5357e926795d009be846687f04556beb 2024-04-29T20:08:25.836 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/ 2024-04-29T20:08:25.836 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3150.g354447ca 2024-04-29T20:08:25.900 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/ 2024-04-29T20:08:25.900 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3150.g354447ca 2024-04-29T20:08:26.024 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-29T20:08:26.024 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:08:26.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-29T20:08:26.049 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-04-29T20:08:26.049 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/354447ca5357e926795d009be846687f04556beb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-29T20:08:26.107 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-04-02-1310/354447ca5357e926795d009be846687f04556beb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-29T20:08:26.108 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:08:26.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-29T20:08:26.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-29T20:08:26.133 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-04-29T20:08:26.134 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/354447ca5357e926795d009be846687f04556beb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-29T20:08:26.186 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-29T20:08:26.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-29T20:08:26.248 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2024-04-29T20:08:26.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2024-04-29T20:08:26.272 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-29T20:08:26.337 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-04-29T20:08:26.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-04-29T20:08:26.474 INFO:teuthology.orchestra.run.smithi167.stdout:48 files removed 2024-04-29T20:08:26.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw 2024-04-29T20:08:26.573 INFO:teuthology.orchestra.run.smithi064.stdout:48 files removed 2024-04-29T20:08:26.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-04-29T20:08:27.238 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 175 kB/s | 84 kB 00:00 2024-04-29T20:08:27.348 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 176 kB/s | 84 kB 00:00 2024-04-29T20:08:27.597 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-29T20:08:27.715 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-29T20:08:27.947 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-29T20:08:28.033 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-04-29T20:08:28.709 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-29T20:08:31.528 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-04-29T20:08:31.852 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-04-29T20:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-29T20:08:40.013 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.2 MB 00:04 2024-04-29T20:08:41.306 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-04-29T20:08:41.386 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 16 kB 00:00 2024-04-29T20:08:42.912 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-29T20:08:43.182 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-29T20:08:44.666 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-29T20:08:48.221 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-29T20:08:49.939 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-29T20:08:50.368 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:19.0.0-3150.g354447ca.el9 ceph 11 M 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 3.4 M 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 3.1 M 2024-04-29T20:08:50.371 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:19.0.0-3150.g354447ca.el9 ceph 5.4 M 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:19.0.0-3150.g354447ca.el9 ceph 21 M 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:19.0.0-3150.g354447ca.el9 ceph 25 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 746 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 500 k 2024-04-29T20:08:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 5.4 M 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse x86_64 2:19.0.0-3150.g354447ca.el9 ceph 45 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:19.0.0-3150.g354447ca.el9 ceph 135 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:19.0.0-3150.g354447ca.el9 ceph 161 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:19.0.0-3150.g354447ca.el9 ceph 320 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:19.0.0-3150.g354447ca.el9 ceph 298 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:19.0.0-3150.g354447ca.el9 ceph 99 k 2024-04-29T20:08:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout:Installing weak dependencies: 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout:Install 30 Packages 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 2 Packages 2024-04-29T20:08:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:08:50.375 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 60 M 2024-04-29T20:08:50.375 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:08:52.163 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:08:52.165 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-3150.g354447ca.el9 ceph 11 M 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 3.4 M 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 3.1 M 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-3150.g354447ca.el9 ceph 5.4 M 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-3150.g354447ca.el9 ceph 21 M 2024-04-29T20:08:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-3150.g354447ca.el9 ceph 25 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 746 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 500 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-3150.g354447ca.el9 ceph 5.4 M 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-29T20:08:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.0.0-3150.g354447ca.el9 ceph 45 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-3150.g354447ca.el9 ceph 135 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-3150.g354447ca.el9 ceph 161 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-3150.g354447ca.el9 ceph 320 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-3150.g354447ca.el9 ceph 298 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-3150.g354447ca.el9 ceph 99 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-29T20:08:52.168 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:08:52.169 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 60 M 2024-04-29T20:08:52.170 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:08:53.930 INFO:teuthology.orchestra.run.smithi167.stdout:(1/32): ceph-base-19.0.0-3150.g354447ca.el9.x86 1.6 MB/s | 5.4 MB 00:03 2024-04-29T20:08:54.030 INFO:teuthology.orchestra.run.smithi167.stdout:(2/32): ceph-selinux-19.0.0-3150.g354447ca.el9. 252 kB/s | 25 kB 00:00 2024-04-29T20:08:54.414 INFO:teuthology.orchestra.run.smithi167.stdout:(3/32): libcephfs2-19.0.0-3150.g354447ca.el9.x8 1.9 MB/s | 746 kB 00:00 2024-04-29T20:08:54.614 INFO:teuthology.orchestra.run.smithi167.stdout:(4/32): libradosstriper1-19.0.0-3150.g354447ca. 2.4 MB/s | 500 kB 00:00 2024-04-29T20:08:55.898 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-base-19.0.0-3150.g354447ca.el9.x86 1.6 MB/s | 5.4 MB 00:03 2024-04-29T20:08:55.998 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-3150.g354447ca.el9. 252 kB/s | 25 kB 00:00 2024-04-29T20:08:56.332 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-3150.g354447ca.el9.x8 2.2 MB/s | 746 kB 00:00 2024-04-29T20:08:56.356 INFO:teuthology.orchestra.run.smithi167.stdout:(5/32): librgw2-19.0.0-3150.g354447ca.el9.x86_6 3.1 MB/s | 5.4 MB 00:01 2024-04-29T20:08:56.473 INFO:teuthology.orchestra.run.smithi167.stdout:(6/32): python3-ceph-argparse-19.0.0-3150.g3544 387 kB/s | 45 kB 00:00 2024-04-29T20:08:56.532 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libradosstriper1-19.0.0-3150.g354447ca. 2.4 MB/s | 500 kB 00:00 2024-04-29T20:08:56.581 INFO:teuthology.orchestra.run.smithi167.stdout:(7/32): python3-ceph-common-19.0.0-3150.g354447 1.2 MB/s | 135 kB 00:00 2024-04-29T20:08:56.698 INFO:teuthology.orchestra.run.smithi167.stdout:(8/32): python3-cephfs-19.0.0-3150.g354447ca.el 1.4 MB/s | 161 kB 00:00 2024-04-29T20:08:56.990 INFO:teuthology.orchestra.run.smithi167.stdout:(9/32): python3-rados-19.0.0-3150.g354447ca.el9 1.1 MB/s | 320 kB 00:00 2024-04-29T20:08:57.166 INFO:teuthology.orchestra.run.smithi167.stdout:(10/32): python3-rbd-19.0.0-3150.g354447ca.el9. 1.7 MB/s | 298 kB 00:00 2024-04-29T20:08:57.266 INFO:teuthology.orchestra.run.smithi167.stdout:(11/32): python3-rgw-19.0.0-3150.g354447ca.el9. 994 kB/s | 99 kB 00:00 2024-04-29T20:08:57.375 INFO:teuthology.orchestra.run.smithi167.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 988 kB/s | 107 kB 00:00 2024-04-29T20:08:57.442 INFO:teuthology.orchestra.run.smithi167.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-04-29T20:08:57.484 INFO:teuthology.orchestra.run.smithi167.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-29T20:08:57.559 INFO:teuthology.orchestra.run.smithi167.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-29T20:08:57.601 INFO:teuthology.orchestra.run.smithi167.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-04-29T20:08:57.634 INFO:teuthology.orchestra.run.smithi167.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-04-29T20:08:57.677 INFO:teuthology.orchestra.run.smithi167.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-29T20:08:57.710 INFO:teuthology.orchestra.run.smithi167.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.1 MB/s | 36 kB 00:00 2024-04-29T20:08:57.744 INFO:teuthology.orchestra.run.smithi167.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-29T20:08:57.841 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): librgw2-19.0.0-3150.g354447ca.el9.x86_6 4.1 MB/s | 5.4 MB 00:01 2024-04-29T20:08:57.933 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.0.0-3150.g3544 493 kB/s | 45 kB 00:00 2024-04-29T20:08:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.0.0-3150.g354447 1.6 MB/s | 135 kB 00:00 2024-04-29T20:08:58.100 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.0.0-3150.g354447ca.el 1.9 MB/s | 161 kB 00:00 2024-04-29T20:08:58.186 INFO:teuthology.orchestra.run.smithi167.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 699 kB/s | 309 kB 00:00 2024-04-29T20:08:58.200 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): python3-rados-19.0.0-3150.g354447ca.el9 3.1 MB/s | 320 kB 00:00 2024-04-29T20:08:58.317 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rbd-19.0.0-3150.g354447ca.el9. 2.5 MB/s | 298 kB 00:00 2024-04-29T20:08:58.409 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rgw-19.0.0-3150.g354447ca.el9. 1.1 MB/s | 99 kB 00:00 2024-04-29T20:08:58.494 INFO:teuthology.orchestra.run.smithi167.stdout:(22/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-29T20:08:58.542 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 803 kB/s | 107 kB 00:00 2024-04-29T20:08:58.561 INFO:teuthology.orchestra.run.smithi167.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 392 kB/s | 26 kB 00:00 2024-04-29T20:08:58.630 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.8 MB/s | 160 kB 00:00 2024-04-29T20:08:58.668 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.2 MB/s | 45 kB 00:00 2024-04-29T20:08:58.752 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-04-29T20:08:58.778 INFO:teuthology.orchestra.run.smithi167.stdout:(24/32): ceph-common-19.0.0-3150.g354447ca.el9. 2.6 MB/s | 21 MB 00:08 2024-04-29T20:08:58.795 INFO:teuthology.orchestra.run.smithi167.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 208 kB/s | 49 kB 00:00 2024-04-29T20:08:58.802 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-29T20:08:58.844 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-04-29T20:08:58.887 INFO:teuthology.orchestra.run.smithi167.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-04-29T20:08:58.894 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-29T20:08:58.936 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 863 kB/s | 36 kB 00:00 2024-04-29T20:08:58.962 INFO:teuthology.orchestra.run.smithi167.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-29T20:08:58.978 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-29T20:08:59.063 INFO:teuthology.orchestra.run.smithi167.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-29T20:08:59.096 INFO:teuthology.orchestra.run.smithi167.stdout:(29/32): libunwind-1.6.2-1.el9.x86_64.rpm 212 kB/s | 67 kB 00:00 2024-04-29T20:08:59.103 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-04-29T20:08:59.229 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): libarrow-9.0.0-11.el9.x86_64.rpm 35 MB/s | 4.4 MB 00:00 2024-04-29T20:08:59.254 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-04-29T20:08:59.287 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-04-29T20:08:59.388 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-29T20:08:59.438 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-04-29T20:08:59.480 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-04-29T20:08:59.538 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-04-29T20:09:00.172 INFO:teuthology.orchestra.run.smithi167.stdout:(30/32): librbd1-19.0.0-3150.g354447ca.el9.x86_ 2.9 MB/s | 3.1 MB 00:01 2024-04-29T20:09:00.280 INFO:teuthology.orchestra.run.smithi167.stdout:(31/32): librados2-19.0.0-3150.g354447ca.el9.x8 2.8 MB/s | 3.4 MB 00:01 2024-04-29T20:09:00.414 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): ceph-radosgw-19.0.0-3150.g354447ca.el9 1.3 MB/s | 11 MB 00:07 2024-04-29T20:09:00.472 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): librados2-19.0.0-3150.g354447ca.el9.x8 3.6 MB/s | 3.4 MB 00:00 2024-04-29T20:09:01.372 INFO:teuthology.orchestra.run.smithi167.stdout:(32/32): ceph-radosgw-19.0.0-3150.g354447ca.el9 1.0 MB/s | 11 MB 00:10 2024-04-29T20:09:01.378 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:01.378 INFO:teuthology.orchestra.run.smithi167.stdout:Total 5.5 MB/s | 60 MB 00:11 2024-04-29T20:09:01.479 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:09:01.516 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:09:01.516 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:09:01.539 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-3150.g354447ca.el9.x86_ 2.8 MB/s | 3.1 MB 00:01 2024-04-29T20:09:01.984 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:09:01.984 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:09:03.503 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:09:03.589 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-29T20:09:03.618 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-29T20:09:03.644 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-29T20:09:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-29T20:09:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-29T20:09:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-29T20:09:03.913 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-29T20:09:03.914 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): ceph-common-19.0.0-3150.g354447ca.el9. 1.8 MB/s | 21 MB 00:11 2024-04-29T20:09:03.917 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:03.918 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.1 MB/s | 60 MB 00:11 2024-04-29T20:09:04.023 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:09:04.064 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:09:04.064 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:09:04.238 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-29T20:09:04.346 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:19.0.0-3150.g354447ca.el9.x86_64 7/34 2024-04-29T20:09:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:19.0.0-3150.g354447ca.el9.x86_64 7/34 2024-04-29T20:09:04.537 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 8/34 2024-04-29T20:09:04.556 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:09:04.557 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:09:04.564 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 9/34 2024-04-29T20:09:04.627 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 9/34 2024-04-29T20:09:04.656 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-29T20:09:04.700 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:19.0.0-3150.g354447ca.el9. 11/34 2024-04-29T20:09:04.778 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 12/34 2024-04-29T20:09:04.800 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_6 13/34 2024-04-29T20:09:04.856 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_6 13/34 2024-04-29T20:09:04.890 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-29T20:09:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-29T20:09:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-04-29T20:09:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-29T20:09:05.098 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-29T20:09:05.127 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-04-29T20:09:05.160 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-29T20:09:05.481 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-29T20:09:05.523 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 22/34 2024-04-29T20:09:05.612 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 22/34 2024-04-29T20:09:05.644 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 23/34 2024-04-29T20:09:05.680 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-29T20:09:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-04-29T20:09:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 26/34 2024-04-29T20:09:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:09:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 26/34 2024-04-29T20:09:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-29T20:09:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-29T20:09:06.341 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 27/34 2024-04-29T20:09:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-29T20:09:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:19.0.0-3150.g354447ca.el9.x8 28/34 2024-04-29T20:09:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-29T20:09:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-29T20:09:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-29T20:09:06.657 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-29T20:09:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-29T20:09:07.104 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-3150.g354447ca.el9.x86_64 7/34 2024-04-29T20:09:07.322 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-3150.g354447ca.el9.x86_64 7/34 2024-04-29T20:09:07.420 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 8/34 2024-04-29T20:09:07.448 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 9/34 2024-04-29T20:09:07.510 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 9/34 2024-04-29T20:09:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-29T20:09:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-3150.g354447ca.el9. 11/34 2024-04-29T20:09:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 12/34 2024-04-29T20:09:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_6 13/34 2024-04-29T20:09:07.747 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_6 13/34 2024-04-29T20:09:07.773 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-29T20:09:07.885 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-29T20:09:07.916 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-04-29T20:09:07.956 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-29T20:09:07.981 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-29T20:09:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-04-29T20:09:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-29T20:09:08.371 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-29T20:09:08.413 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 22/34 2024-04-29T20:09:08.507 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 22/34 2024-04-29T20:09:08.538 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 23/34 2024-04-29T20:09:08.573 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-29T20:09:08.690 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 29/34 2024-04-29T20:09:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 29/34 2024-04-29T20:09:09.078 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-04-29T20:09:09.102 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 26/34 2024-04-29T20:09:09.165 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 26/34 2024-04-29T20:09:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 27/34 2024-04-29T20:09:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 29/34 2024-04-29T20:09:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-3150.g354447ca.el9.x8 28/34 2024-04-29T20:09:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 30/34 2024-04-29T20:09:09.402 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 30/34 2024-04-29T20:09:09.402 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-29T20:09:09.403 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-29T20:09:09.403 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:09.430 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 31/34 2024-04-29T20:09:11.724 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 29/34 2024-04-29T20:09:11.758 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 29/34 2024-04-29T20:09:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 29/34 2024-04-29T20:09:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 30/34 2024-04-29T20:09:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 30/34 2024-04-29T20:09:12.463 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-29T20:09:12.464 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-29T20:09:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 31/34 2024-04-29T20:09:18.804 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 31/34 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /root 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2024-04-29T20:09:18.805 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:19.773 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 32/34 2024-04-29T20:09:19.798 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 32/34 2024-04-29T20:09:19.798 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:19.798 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-29T20:09:19.798 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-29T20:09:19.798 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-29T20:09:19.798 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:19.800 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-29T20:09:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-29T20:09:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 1/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 2/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 3/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 4/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 5/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_6 6/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 7/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3150.g354447ca.el9. 8/34 2024-04-29T20:09:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:19.0.0-3150.g354447ca.el9.x8 9/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 10/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 11/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 12/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 13/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-29T20:09:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:19.0.0-3150.g354447ca.el9.x86_64 31/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-29T20:09:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 33/34 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-29T20:09:21.889 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-29T20:09:21.890 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-29T20:09:21.891 INFO:teuthology.orchestra.run.smithi167.stdout: re2-1:20211101-3.el9.x86_64 2024-04-29T20:09:21.892 INFO:teuthology.orchestra.run.smithi167.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-29T20:09:21.892 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:21.892 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:22.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test 2024-04-29T20:09:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 31/34 2024-04-29T20:09:22.193 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-04-29T20:09:22.193 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-04-29T20:09:22.194 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-04-29T20:09:22.194 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-04-29T20:09:22.194 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-04-29T20:09:22.194 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-04-29T20:09:22.194 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-04-29T20:09:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:22.703 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:34 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:22.829 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:19.0.0-3150.g354447ca.el9 ceph 49 M 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:19.0.0-3150.g354447ca.el9 ceph 163 k 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:09:22.830 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:22.831 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-04-29T20:09:22.831 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:22.831 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 50 M 2024-04-29T20:09:22.831 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 211 M 2024-04-29T20:09:22.831 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:09:23.204 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 32/34 2024-04-29T20:09:23.209 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): libcephsqlite-19.0.0-3150.g354447ca.el9. 652 kB/s | 163 kB 00:00 2024-04-29T20:09:23.236 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 32/34 2024-04-29T20:09:23.236 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:23.236 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-29T20:09:23.236 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-29T20:09:23.236 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-29T20:09:23.236 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:23.238 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-29T20:09:23.268 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 986 kB/s | 305 kB 00:00 2024-04-29T20:09:23.319 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 584 kB/s | 64 kB 00:00 2024-04-29T20:09:23.323 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-29T20:09:23.323 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 1/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 2/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 3/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 4/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 5/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_6 6/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 7/34 2024-04-29T20:09:24.504 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3150.g354447ca.el9. 8/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-3150.g354447ca.el9.x8 9/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 10/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 11/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 12/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 13/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-29T20:09:24.505 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-3150.g354447ca.el9.x86_64 31/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-29T20:09:24.506 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 33/34 2024-04-29T20:09:25.262 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-29T20:09:25.263 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:25.265 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:25.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-04-29T20:09:25.697 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): ceph-test-19.0.0-3150.g354447ca.el9.x86_ 18 MB/s | 49 MB 00:02 2024-04-29T20:09:25.700 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:25.701 INFO:teuthology.orchestra.run.smithi167.stdout:Total 17 MB/s | 50 MB 00:02 2024-04-29T20:09:25.728 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:09:25.749 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:09:25.749 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:09:26.078 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:37 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:26.121 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:09:26.122 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:09:26.201 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-3150.g354447ca.el9 ceph 49 M 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-3150.g354447ca.el9 ceph 163 k 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:26.202 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:09:26.203 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:26.203 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-04-29T20:09:26.203 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:26.203 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 50 M 2024-04-29T20:09:26.204 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 211 M 2024-04-29T20:09:26.204 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:09:26.404 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-04-29T20:09:26.446 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-29T20:09:26.497 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): libcephsqlite-19.0.0-3150.g354447ca.el9. 658 kB/s | 163 kB 00:00 2024-04-29T20:09:26.651 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:09:26.739 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-29T20:09:26.788 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-29T20:09:26.815 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 3/4 2024-04-29T20:09:28.768 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-19.0.0-3150.g354447ca.el9.x86_ 19 MB/s | 49 MB 00:02 2024-04-29T20:09:28.770 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:28.770 INFO:teuthology.orchestra.run.smithi064.stdout:Total 19 MB/s | 50 MB 00:02 2024-04-29T20:09:28.797 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:09:28.820 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:09:28.820 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:09:29.211 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:09:29.212 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:09:29.753 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:09:29.836 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-29T20:09:29.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-29T20:09:29.902 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 3/4 2024-04-29T20:09:31.444 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 3/4 2024-04-29T20:09:31.512 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 4/4 2024-04-29T20:09:33.923 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 4/4 2024-04-29T20:09:33.923 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 1/4 2024-04-29T20:09:33.923 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 2/4 2024-04-29T20:09:33.923 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-29T20:09:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:34.204 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:34.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph 2024-04-29T20:09:34.769 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 3/4 2024-04-29T20:09:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 4/4 2024-04-29T20:09:35.061 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:47 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:35.190 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:19.0.0-3150.g354447ca.el9 ceph 6.6 k 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:19.0.0-3150.g354447ca.el9 ceph 2.4 M 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.5 M 2024-04-29T20:09:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 249 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:19.0.0-3150.g354447ca.el9 ceph 4.7 M 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:19.0.0-3150.g354447ca.el9 ceph 17 M 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-29T20:09:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-29T20:09:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout:Install 36 Packages 2024-04-29T20:09:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:35.197 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 30 M 2024-04-29T20:09:35.197 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 107 M 2024-04-29T20:09:35.197 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:09:35.999 INFO:teuthology.orchestra.run.smithi167.stdout:(1/36): ceph-19.0.0-3150.g354447ca.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-04-29T20:09:36.241 INFO:teuthology.orchestra.run.smithi167.stdout:(2/36): ceph-mgr-19.0.0-3150.g354447ca.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-04-29T20:09:36.300 INFO:teuthology.orchestra.run.smithi167.stdout:(3/36): ceph-mds-19.0.0-3150.g354447ca.el9.x86_ 5.1 MB/s | 2.4 MB 00:00 2024-04-29T20:09:36.374 INFO:teuthology.orchestra.run.smithi167.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3150.g3544 3.3 MB/s | 249 kB 00:00 2024-04-29T20:09:36.825 INFO:teuthology.orchestra.run.smithi167.stdout:(5/36): ceph-mon-19.0.0-3150.g354447ca.el9.x86_ 5.7 MB/s | 4.7 MB 00:00 2024-04-29T20:09:36.967 INFO:teuthology.orchestra.run.smithi167.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 428 kB/s | 253 kB 00:00 2024-04-29T20:09:37.125 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 4/4 2024-04-29T20:09:37.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 1/4 2024-04-29T20:09:37.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 2/4 2024-04-29T20:09:37.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-29T20:09:37.150 INFO:teuthology.orchestra.run.smithi167.stdout:(7/36): ceph-osd-19.0.0-3150.g354447ca.el9.x86_ 19 MB/s | 17 MB 00:00 2024-04-29T20:09:37.176 INFO:teuthology.orchestra.run.smithi167.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-04-29T20:09:37.284 INFO:teuthology.orchestra.run.smithi167.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-04-29T20:09:37.393 INFO:teuthology.orchestra.run.smithi167.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:37.415 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:37.576 INFO:teuthology.orchestra.run.smithi167.stdout:(11/36): python3-pycparser-2.20-6.el9.noarch.rp 318 kB/s | 135 kB 00:00 2024-04-29T20:09:37.602 INFO:teuthology.orchestra.run.smithi167.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 904 kB/s | 188 kB 00:00 2024-04-29T20:09:37.660 INFO:teuthology.orchestra.run.smithi167.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 1.5 MB/s | 1.2 MB 00:00 2024-04-29T20:09:37.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-04-29T20:09:37.686 INFO:teuthology.orchestra.run.smithi167.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 415 kB/s | 35 kB 00:00 2024-04-29T20:09:37.744 INFO:teuthology.orchestra.run.smithi167.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 716 kB/s | 42 kB 00:00 2024-04-29T20:09:37.803 INFO:teuthology.orchestra.run.smithi167.stdout:(16/36): python3-mako-1.1.4-6.el9.noarch.rpm 761 kB/s | 172 kB 00:00 2024-04-29T20:09:37.828 INFO:teuthology.orchestra.run.smithi167.stdout:(17/36): python3-pytz-2021.1-5.el9.noarch.rpm 304 kB/s | 51 kB 00:00 2024-04-29T20:09:37.944 INFO:teuthology.orchestra.run.smithi167.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-04-29T20:09:37.979 INFO:teuthology.orchestra.run.smithi167.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-04-29T20:09:38.088 INFO:teuthology.orchestra.run.smithi167.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 532 kB/s | 151 kB 00:00 2024-04-29T20:09:38.113 INFO:teuthology.orchestra.run.smithi167.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-04-29T20:09:38.138 INFO:teuthology.orchestra.run.smithi167.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-04-29T20:09:38.164 INFO:teuthology.orchestra.run.smithi167.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-04-29T20:09:38.189 INFO:teuthology.orchestra.run.smithi167.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-04-29T20:09:38.200 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:49 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:38.214 INFO:teuthology.orchestra.run.smithi167.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-04-29T20:09:38.239 INFO:teuthology.orchestra.run.smithi167.stdout:(26/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 585 kB/s | 172 kB 00:00 2024-04-29T20:09:38.265 INFO:teuthology.orchestra.run.smithi167.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-04-29T20:09:38.290 INFO:teuthology.orchestra.run.smithi167.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-04-29T20:09:38.315 INFO:teuthology.orchestra.run.smithi167.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-29T20:09:38.333 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:38.336 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:09:38.336 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:38.336 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-3150.g354447ca.el9 ceph 6.6 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-3150.g354447ca.el9 ceph 2.4 M 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.5 M 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 249 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-3150.g354447ca.el9 ceph 4.7 M 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-3150.g354447ca.el9 ceph 17 M 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-29T20:09:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-29T20:09:38.338 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:38.339 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:09:38.340 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:38.340 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-04-29T20:09:38.340 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:38.340 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-04-29T20:09:38.340 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-04-29T20:09:38.340 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:09:38.341 INFO:teuthology.orchestra.run.smithi167.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-04-29T20:09:38.365 INFO:teuthology.orchestra.run.smithi167.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-29T20:09:38.391 INFO:teuthology.orchestra.run.smithi167.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-04-29T20:09:38.416 INFO:teuthology.orchestra.run.smithi167.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-29T20:09:38.441 INFO:teuthology.orchestra.run.smithi167.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-29T20:09:38.466 INFO:teuthology.orchestra.run.smithi167.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-04-29T20:09:38.492 INFO:teuthology.orchestra.run.smithi167.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-04-29T20:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:38.495 INFO:teuthology.orchestra.run.smithi167.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-04-29T20:09:38.633 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:09:38.673 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:09:38.674 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:09:38.971 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:09:38.972 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:09:39.076 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-3150.g354447ca.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-04-29T20:09:39.310 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mds-19.0.0-3150.g354447ca.el9.x86_ 5.7 MB/s | 2.4 MB 00:00 2024-04-29T20:09:39.477 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mgr-19.0.0-3150.g354447ca.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-04-29T20:09:39.702 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3150.g3544 1.1 MB/s | 249 kB 00:00 2024-04-29T20:09:39.769 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): ceph-mon-19.0.0-3150.g354447ca.el9.x86_ 6.8 MB/s | 4.7 MB 00:00 2024-04-29T20:09:39.953 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-04-29T20:09:40.019 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-29T20:09:40.222 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:09:40.236 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): ceph-osd-19.0.0-3150.g354447ca.el9.x86_ 18 MB/s | 17 MB 00:00 2024-04-29T20:09:40.262 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 558 kB/s | 135 kB 00:00 2024-04-29T20:09:40.278 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-29T20:09:40.310 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-29T20:09:40.337 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 2.2 MB/s | 1.2 MB 00:00 2024-04-29T20:09:40.338 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-29T20:09:40.395 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.6 MB/s | 215 kB 00:00 2024-04-29T20:09:40.410 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-29T20:09:40.437 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): python3-requests-2.25.1-8.el9.noarch.r 623 kB/s | 125 kB 00:00 2024-04-29T20:09:40.462 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-04-29T20:09:40.471 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-29T20:09:40.504 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-29T20:09:40.504 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-29T20:09:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-29T20:09:40.646 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 166 kB/s | 35 kB 00:00 2024-04-29T20:09:40.672 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): python3-mako-1.1.4-6.el9.noarch.rpm 624 kB/s | 172 kB 00:00 2024-04-29T20:09:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-29T20:09:40.697 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 217 kB/s | 42 kB 00:00 2024-04-29T20:09:40.715 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-29T20:09:40.750 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-29T20:09:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-29T20:09:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-29T20:09:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-29T20:09:40.880 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 826 kB/s | 151 kB 00:00 2024-04-29T20:09:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-29T20:09:40.906 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 209 kB/s | 43 kB 00:00 2024-04-29T20:09:40.923 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-29T20:09:40.948 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-29T20:09:40.958 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-29T20:09:40.973 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): lua-devel-5.4.4-4.el9.x86_64.rpm 68 kB/s | 22 kB 00:00 2024-04-29T20:09:40.999 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-04-29T20:09:41.004 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-29T20:09:41.023 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-04-29T20:09:41.049 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-04-29T20:09:41.074 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-04-29T20:09:41.077 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-29T20:09:41.099 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-04-29T20:09:41.124 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-04-29T20:09:41.149 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-04-29T20:09:41.175 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-29T20:09:41.200 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-29T20:09:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-29T20:09:41.225 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 331 kB/s | 16 kB 00:00 2024-04-29T20:09:41.250 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-29T20:09:41.267 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 537 kB/s | 36 kB 00:00 2024-04-29T20:09:41.297 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-29T20:09:41.301 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-29T20:09:41.326 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-04-29T20:09:41.343 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-04-29T20:09:41.347 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:41.348 INFO:teuthology.orchestra.run.smithi064.stdout:Total 10 MB/s | 30 MB 00:03 2024-04-29T20:09:41.357 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-29T20:09:41.417 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-29T20:09:41.470 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-29T20:09:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:09:41.525 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-29T20:09:41.533 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:09:41.533 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:09:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-29T20:09:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-29T20:09:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-29T20:09:41.845 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:09:41.847 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:09:41.925 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-29T20:09:41.958 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-29T20:09:42.075 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-29T20:09:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3150.g354447ca.el9. 31/36 2024-04-29T20:09:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 32/36 2024-04-29T20:09:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 32/36 2024-04-29T20:09:42.285 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:42.285 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-29T20:09:42.285 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-29T20:09:42.285 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-29T20:09:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:09:43.180 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-29T20:09:43.212 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-29T20:09:43.241 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-29T20:09:43.317 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-29T20:09:43.356 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-29T20:09:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-29T20:09:43.486 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-29T20:09:43.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-29T20:09:43.617 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-29T20:09:43.652 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-29T20:09:43.681 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-29T20:09:43.715 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-29T20:09:43.728 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 33/36 2024-04-29T20:09:43.749 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-29T20:09:43.754 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 33/36 2024-04-29T20:09:43.754 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:43.754 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-29T20:09:43.754 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-29T20:09:43.754 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-29T20:09:43.754 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:43.790 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-29T20:09:43.842 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-29T20:09:43.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-29T20:09:43.930 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-29T20:09:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-29T20:09:44.161 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-29T20:09:44.212 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 34/36 2024-04-29T20:09:44.235 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 34/36 2024-04-29T20:09:44.235 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:44.235 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-29T20:09:44.235 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-29T20:09:44.235 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-29T20:09:44.235 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-29T20:09:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-29T20:09:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-29T20:09:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-29T20:09:44.463 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 35/36 2024-04-29T20:09:44.486 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 35/36 2024-04-29T20:09:44.486 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:44.486 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-29T20:09:44.486 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-29T20:09:44.486 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-29T20:09:44.486 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-29T20:09:44.549 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:19.0.0-3150.g354447ca.el9.x86_64 36/36 2024-04-29T20:09:44.686 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-29T20:09:44.743 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-29T20:09:44.825 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-29T20:09:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-29T20:09:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-29T20:09:45.093 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-29T20:09:45.263 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3150.g354447ca.el9. 31/36 2024-04-29T20:09:45.293 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 32/36 2024-04-29T20:09:45.319 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 32/36 2024-04-29T20:09:45.320 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:45.320 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-29T20:09:45.320 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-29T20:09:45.320 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-29T20:09:45.320 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-2:19.0.0-3150.g354447ca.el9.x86_64 36/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:19.0.0-3150.g354447ca.el9.x86_64 1/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 2/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 3/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 4/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 5/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3150.g354447ca.el9. 6/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-29T20:09:46.317 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-29T20:09:46.318 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-29T20:09:46.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-29T20:09:46.835 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 33/36 2024-04-29T20:09:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 33/36 2024-04-29T20:09:46.863 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:46.863 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-29T20:09:46.863 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-29T20:09:46.863 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-29T20:09:46.863 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 34/36 2024-04-29T20:09:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 34/36 2024-04-29T20:09:47.352 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:47.352 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-29T20:09:47.352 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-29T20:09:47.352 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-29T20:09:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:47.372 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: lua-5.4.4-4.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-29T20:09:47.373 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply-3.11-14.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-29T20:09:47.374 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:47.375 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:47.586 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 35/36 2024-04-29T20:09:47.611 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 35/36 2024-04-29T20:09:47.611 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:47.611 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-29T20:09:47.611 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-29T20:09:47.611 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-29T20:09:47.611 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:47.668 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-3150.g354447ca.el9.x86_64 36/36 2024-04-29T20:09:47.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base 2024-04-29T20:09:48.258 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:00 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:48.337 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:09:48.378 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:48.379 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:09:48.379 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:48.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install cephadm 2024-04-29T20:09:48.948 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:00 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:49.070 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 762 k 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:09:49.071 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:49.072 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 762 k 2024-04-29T20:09:49.072 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 769 k 2024-04-29T20:09:49.072 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:09:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-3150.g354447ca.el9.x86_64 36/36 2024-04-29T20:09:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-3150.g354447ca.el9.x86_64 1/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 2/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 3/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 4/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 5/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3150.g354447ca.el9. 6/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-29T20:09:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-29T20:09:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-29T20:09:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-29T20:09:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-29T20:09:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-29T20:09:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-29T20:09:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-29T20:09:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-29T20:09:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-29T20:09:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-29T20:09:49.388 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm-19.0.0-3150.g354447ca.el9.noarch.rpm 2.4 MB/s | 762 kB 00:00 2024-04-29T20:09:49.389 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:49.389 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.3 MB/s | 762 kB 00:00 2024-04-29T20:09:49.390 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:09:49.394 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:09:49.394 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:09:49.401 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:09:49.401 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:09:49.469 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:09:49.910 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:50.272 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-29T20:09:50.272 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:50.272 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:09:50.272 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:50.272 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-29T20:09:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-29T20:09:50.274 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:50.275 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:50.529 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:09:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:09:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:50.529 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:50.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-04-29T20:09:50.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache 2024-04-29T20:09:51.131 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:02 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:51.213 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:09:51.256 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:51.258 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:09:51.258 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:51.273 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:03 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:51.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-04-29T20:09:51.394 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:51.395 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:09:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:09:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3150.g354447ca.el9 ceph 145 k 2024-04-29T20:09:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 145 k 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 435 k 2024-04-29T20:09:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:09:51.595 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-19.0.0-3150.g354447 728 kB/s | 145 kB 00:00 2024-04-29T20:09:51.596 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:51.596 INFO:teuthology.orchestra.run.smithi167.stdout:Total 720 kB/s | 145 kB 00:00 2024-04-29T20:09:51.597 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:09:51.604 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:09:51.604 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:09:51.661 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:09:51.661 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:09:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:09:51.847 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:02 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:51.896 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3150.g354447ca. 1/1 2024-04-29T20:09:51.921 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3150.g354447ca. 1/1 2024-04-29T20:09:51.921 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:51.921 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-29T20:09:51.922 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:51.973 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 762 k 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:51.974 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:09:51.975 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:51.975 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 762 k 2024-04-29T20:09:51.975 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 769 k 2024-04-29T20:09:51.975 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:09:52.328 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-3150.g354447ca.el9.noarch.rpm 2.1 MB/s | 762 kB 00:00 2024-04-29T20:09:52.328 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:52.329 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.1 MB/s | 762 kB 00:00 2024-04-29T20:09:52.329 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:09:52.334 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:09:52.334 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:09:52.340 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:09:52.341 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:09:52.418 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:09:52.797 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3150.g354447ca. 1/1 2024-04-29T20:09:52.798 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:52.798 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:09:52.798 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:52.798 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:52.798 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:52.878 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:52.983 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:53.000 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr 2024-04-29T20:09:53.276 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:53.495 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/1 2024-04-29T20:09:53.496 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:53.496 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:09:53.496 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:09:53.496 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:53.496 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:53.518 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:05 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:53.603 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:09:53.646 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:53.647 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:09:53.647 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:09:53.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-dashboard 2024-04-29T20:09:53.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-04-29T20:09:54.216 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:06 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:09:54.276 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:05 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:54.337 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 3.5 M 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 34 k 2024-04-29T20:09:54.338 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 16 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-04-29T20:09:54.339 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:09:54.340 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 8.6 M 2024-04-29T20:09:54.340 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 104 M 2024-04-29T20:09:54.340 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:09:54.400 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3150.g354447ca.el9 ceph 145 k 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:09:54.401 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:54.402 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-04-29T20:09:54.402 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 435 k 2024-04-29T20:09:54.402 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:09:54.625 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-3150.g354447 649 kB/s | 145 kB 00:00 2024-04-29T20:09:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:Total 645 kB/s | 145 kB 00:00 2024-04-29T20:09:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:09:54.634 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:09:54.635 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:09:54.694 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:09:54.695 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:09:54.799 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3150.g35 179 kB/s | 34 kB 00:00 2024-04-29T20:09:54.840 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:09:54.899 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3150.g354 56 kB/s | 16 kB 00:00 2024-04-29T20:09:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3150.g354447ca. 1/1 2024-04-29T20:09:54.977 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3150.g354447ca. 1/1 2024-04-29T20:09:54.977 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:09:54.977 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-29T20:09:54.977 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:55.082 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-04-29T20:09:55.124 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-04-29T20:09:55.158 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 11 MB/s | 863 kB 00:00 2024-04-29T20:09:55.392 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-04-29T20:09:55.642 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-04-29T20:09:55.700 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): ceph-mgr-dashboard-19.0.0-3150.g354447c 3.2 MB/s | 3.5 MB 00:01 2024-04-29T20:09:55.742 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-04-29T20:09:55.851 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-04-29T20:09:55.872 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3150.g354447ca. 1/1 2024-04-29T20:09:55.872 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:55.872 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:09:55.872 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:09:55.872 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:55.872 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:55.926 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 136 kB/s | 31 kB 00:00 2024-04-29T20:09:55.968 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-04-29T20:09:55.971 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:55.971 INFO:teuthology.orchestra.run.smithi167.stdout:Total 5.3 MB/s | 8.6 MB 00:01 2024-04-29T20:09:56.018 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:09:56.035 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:09:56.035 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:09:56.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-04-29T20:09:56.191 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:09:56.191 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:09:56.602 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:07 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:56.685 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:09:56.723 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:09:56.728 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:56.729 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:09:56.730 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:09:56.786 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-29T20:09:56.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-04-29T20:09:56.821 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-29T20:09:56.867 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-29T20:09:56.898 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-29T20:09:56.975 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-29T20:09:57.100 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-29T20:09:57.193 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-29T20:09:57.236 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-29T20:09:57.264 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-29T20:09:57.297 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3150.g354447ca.el9 10/12 2024-04-29T20:09:57.318 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:08 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:09:57.444 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 3.5 M 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 34 k 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 16 k 2024-04-29T20:09:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-04-29T20:09:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:09:57.447 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 8.6 M 2024-04-29T20:09:57.447 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 104 M 2024-04-29T20:09:57.447 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:09:58.162 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3150.g35 170 kB/s | 34 kB 00:00 2024-04-29T20:09:58.262 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3150.g354 54 kB/s | 16 kB 00:00 2024-04-29T20:09:58.346 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-04-29T20:09:58.354 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3150.g354447ca.el 11/12 2024-04-29T20:09:58.389 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noa 12/12 2024-04-29T20:09:58.412 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-04-29T20:09:58.486 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): ceph-mgr-dashboard-19.0.0-3150.g354447c 6.7 MB/s | 3.5 MB 00:00 2024-04-29T20:09:58.813 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.8 MB/s | 863 kB 00:00 2024-04-29T20:09:58.896 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 47 kB/s | 19 kB 00:00 2024-04-29T20:09:59.188 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 706 kB/s | 548 kB 00:00 2024-04-29T20:09:59.238 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 456 kB/s | 144 kB 00:00 2024-04-29T20:09:59.289 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 307 kB/s | 31 kB 00:00 2024-04-29T20:09:59.431 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 981 kB/s | 188 kB 00:00 2024-04-29T20:09:59.739 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-04-29T20:09:59.746 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:09:59.746 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.8 MB/s | 8.6 MB 00:02 2024-04-29T20:09:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noa 12/12 2024-04-29T20:09:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3150.g354447ca.el 1/12 2024-04-29T20:09:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noa 2/12 2024-04-29T20:09:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3150.g354447ca.el9 3/12 2024-04-29T20:09:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-29T20:09:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-29T20:09:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-29T20:09:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-29T20:09:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-29T20:09:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-29T20:09:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-29T20:09:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-29T20:09:59.794 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:09:59.813 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:09:59.813 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:09:59.976 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:09:59.977 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:00.079 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-29T20:10:00.079 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:00.079 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:00.079 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-29T20:10:00.079 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:00.079 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:00.080 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:00.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-29T20:10:00.514 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-29T20:10:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-29T20:10:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-29T20:10:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-29T20:10:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-29T20:10:00.857 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:12 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-29T20:10:00.978 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 7.4 M 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:10:00.979 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-04-29T20:10:00.980 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-04-29T20:10:00.981 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:00.981 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 38 M 2024-04-29T20:10:00.981 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 209 M 2024-04-29T20:10:00.981 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:01.001 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-29T20:10:01.050 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-29T20:10:01.077 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-29T20:10:01.110 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3150.g354447ca.el9 10/12 2024-04-29T20:10:01.952 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-31 12 MB/s | 7.4 MB 00:00 2024-04-29T20:10:02.010 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 563 kB/s | 33 kB 00:00 2024-04-29T20:10:02.188 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-04-29T20:10:02.221 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3150.g354447ca.el 11/12 2024-04-29T20:10:02.238 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-04-29T20:10:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noa 12/12 2024-04-29T20:10:02.289 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-29T20:10:02.347 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): libquadmath-11.4.1-3.el9.x86_64.rpm 190 kB/s | 195 kB 00:01 2024-04-29T20:10:02.422 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-29T20:10:02.548 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-04-29T20:10:02.631 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-04-29T20:10:02.715 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-29T20:10:03.190 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-29T20:10:03.257 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 414 kB/s | 803 kB 00:01 2024-04-29T20:10:03.258 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:03.258 INFO:teuthology.orchestra.run.smithi167.stdout:Total 17 MB/s | 38 MB 00:02 2024-04-29T20:10:03.523 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noa 12/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3150.g354447ca.el 1/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noa 2/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3150.g354447ca.el9 3/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-29T20:10:03.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-29T20:10:03.610 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:03.638 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:03.638 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:03.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-29T20:10:03.819 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-29T20:10:03.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-29T20:10:03.821 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:03.821 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:04.047 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:04.047 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:04.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-29T20:10:04.591 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:15 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:04.718 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:04.718 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-29T20:10:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 7.4 M 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-29T20:10:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-04-29T20:10:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:04.721 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-04-29T20:10:04.721 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-04-29T20:10:04.721 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:04.892 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:04.953 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-29T20:10:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-29T20:10:05.091 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-29T20:10:05.113 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 939 kB/s | 195 kB 00:00 2024-04-29T20:10:05.140 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-29T20:10:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-04-29T20:10:05.189 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-04-29T20:10:05.231 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-04-29T20:10:05.273 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 35 kB 00:00 2024-04-29T20:10:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-29T20:10:05.350 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-04-29T20:10:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-29T20:10:05.432 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-29T20:10:05.440 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-04-29T20:10:05.598 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-04-29T20:10:05.674 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-04-29T20:10:05.741 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-29T20:10:06.111 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-29T20:10:06.192 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-04-29T20:10:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-29T20:10:07.470 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-29T20:10:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-29T20:10:08.547 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3150.g35444 12/12 2024-04-29T20:10:09.274 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 1.7 MB/s | 7.4 MB 00:04 2024-04-29T20:10:09.275 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:09.276 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.3 MB/s | 38 MB 00:04 2024-04-29T20:10:09.641 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:09.668 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:09.668 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:10.106 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:10.106 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-29T20:10:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-29T20:10:11.179 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-29T20:10:11.225 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-29T20:10:11.240 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3150.g35444 12/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3150.g35444 1/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-29T20:10:11.241 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-29T20:10:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-29T20:10:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-29T20:10:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-29T20:10:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-29T20:10:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-29T20:10:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-29T20:10:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:11.647 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:12.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-rook 2024-04-29T20:10:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-29T20:10:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-29T20:10:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:24 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:12.679 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:12.680 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:12.680 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:10:12.680 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:12.680 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:12.680 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 49 k 2024-04-29T20:10:12.680 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:12.681 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:12.682 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:12.682 INFO:teuthology.orchestra.run.smithi167.stdout:Install 13 Packages 2024-04-29T20:10:12.682 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:12.682 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 2.2 M 2024-04-29T20:10:12.682 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-04-29T20:10:12.682 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:13.257 INFO:teuthology.orchestra.run.smithi167.stdout:(1/13): ceph-mgr-rook-19.0.0-3150.g354447ca.el9 136 kB/s | 49 kB 00:00 2024-04-29T20:10:13.524 INFO:teuthology.orchestra.run.smithi167.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 30 kB/s | 19 kB 00:00 2024-04-29T20:10:13.574 INFO:teuthology.orchestra.run.smithi167.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 38 kB/s | 26 kB 00:00 2024-04-29T20:10:13.666 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-29T20:10:14.216 INFO:teuthology.orchestra.run.smithi167.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 232 kB/s | 222 kB 00:00 2024-04-29T20:10:14.416 INFO:teuthology.orchestra.run.smithi167.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 183 kB/s | 159 kB 00:00 2024-04-29T20:10:14.600 INFO:teuthology.orchestra.run.smithi167.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 272 kB/s | 279 kB 00:01 2024-04-29T20:10:14.700 INFO:teuthology.orchestra.run.smithi167.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 114 kB/s | 32 kB 00:00 2024-04-29T20:10:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-29T20:10:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3150.g35444 12/12 2024-04-29T20:10:14.850 INFO:teuthology.orchestra.run.smithi167.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-04-29T20:10:14.917 INFO:teuthology.orchestra.run.smithi167.stdout:(9/13): python3-google-auth-2.29.0-1.el9.noarch 947 kB/s | 206 kB 00:00 2024-04-29T20:10:14.967 INFO:teuthology.orchestra.run.smithi167.stdout:(10/13): python3-requests-oauthlib-1.3.0-12.el9 71 kB/s | 54 kB 00:00 2024-04-29T20:10:15.001 INFO:teuthology.orchestra.run.smithi167.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 708 kB/s | 59 kB 00:00 2024-04-29T20:10:15.209 INFO:teuthology.orchestra.run.smithi167.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-04-29T20:10:15.302 INFO:teuthology.orchestra.run.smithi167.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 269 kB/s | 90 kB 00:00 2024-04-29T20:10:15.305 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:15.305 INFO:teuthology.orchestra.run.smithi167.stdout:Total 859 kB/s | 2.2 MB 00:02 2024-04-29T20:10:15.369 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:15.385 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:15.385 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:15.473 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:15.473 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:15.980 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:16.148 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-29T20:10:16.203 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-29T20:10:16.259 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-29T20:10:16.296 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-29T20:10:16.337 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-29T20:10:16.440 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-29T20:10:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-29T20:10:16.597 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-29T20:10:17.334 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-29T20:10:17.386 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3150.g35444 12/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3150.g35444 1/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-29T20:10:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-29T20:10:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-29T20:10:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-29T20:10:17.428 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-29T20:10:17.476 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-29T20:10:17.498 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 13/13 2024-04-29T20:10:17.837 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-29T20:10:17.838 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-29T20:10:17.839 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:17.839 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:18.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 13/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 1/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-29T20:10:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-29T20:10:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-29T20:10:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-29T20:10:18.558 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-29T20:10:18.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-29T20:10:18.560 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-29T20:10:18.560 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-29T20:10:18.560 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-29T20:10:18.560 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:18.560 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:18.754 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:29 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:18.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-cephadm 2024-04-29T20:10:18.881 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 49 k 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:10:18.882 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-29T20:10:18.883 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-04-29T20:10:18.884 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:19.131 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 26 kB 00:00 2024-04-29T20:10:19.156 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 125 kB/s | 19 kB 00:00 2024-04-29T20:10:19.290 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-29T20:10:19.315 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-04-29T20:10:19.322 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:31 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:19.373 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 920 kB/s | 54 kB 00:00 2024-04-29T20:10:19.399 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-04-29T20:10:19.441 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): ceph-mgr-rook-19.0.0-3150.g354447ca.el9 112 kB/s | 49 kB 00:00 2024-04-29T20:10:19.446 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:19.447 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:19.447 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:10:19.447 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:19.447 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 142 k 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-04-29T20:10:19.448 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:19.449 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 7.0 M 2024-04-29T20:10:19.449 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 33 M 2024-04-29T20:10:19.449 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:19.849 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 33 kB/s | 15 kB 00:00 2024-04-29T20:10:19.908 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 60 kB/s | 32 kB 00:00 2024-04-29T20:10:20.100 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-29T20:10:20.225 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 262 kB/s | 206 kB 00:00 2024-04-29T20:10:20.230 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3150.g354447ca.e 300 kB/s | 142 kB 00:00 2024-04-29T20:10:20.292 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 467 kB/s | 90 kB 00:00 2024-04-29T20:10:20.503 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-04-29T20:10:20.571 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 863 kB/s | 58 kB 00:00 2024-04-29T20:10:20.621 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-04-29T20:10:20.771 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 244 kB/s | 248 kB 00:01 2024-04-29T20:10:21.242 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 751 kB/s | 1.0 MB 00:01 2024-04-29T20:10:21.246 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:21.246 INFO:teuthology.orchestra.run.smithi064.stdout:Total 954 kB/s | 2.2 MB 00:02 2024-04-29T20:10:21.317 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:21.334 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:21.334 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:21.431 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:21.431 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:21.959 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:22.142 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-29T20:10:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-29T20:10:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-29T20:10:22.286 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-29T20:10:22.329 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-29T20:10:22.426 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-29T20:10:22.554 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-29T20:10:22.605 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-29T20:10:23.380 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-29T20:10:23.442 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-29T20:10:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-29T20:10:23.534 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-29T20:10:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 13/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 13/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 1/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-29T20:10:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-29T20:10:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-29T20:10:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-29T20:10:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-29T20:10:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-29T20:10:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-29T20:10:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-29T20:10:24.619 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-29T20:10:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-29T20:10:24.620 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:24.620 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:24.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-04-29T20:10:25.383 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:36 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:25.509 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:25.510 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:25.510 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:10:25.510 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:25.510 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:25.510 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 142 k 2024-04-29T20:10:25.510 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-04-29T20:10:25.511 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:25.512 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-04-29T20:10:25.512 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-04-29T20:10:25.512 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:25.910 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-29T20:10:26.052 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-04-29T20:10:26.177 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 463 kB/s | 58 kB 00:00 2024-04-29T20:10:26.236 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 871 kB/s | 51 kB 00:00 2024-04-29T20:10:26.261 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-04-29T20:10:26.395 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): ceph-mgr-cephadm-19.0.0-3150.g354447ca.e 213 kB/s | 142 kB 00:00 2024-04-29T20:10:26.396 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:26.396 INFO:teuthology.orchestra.run.smithi064.stdout:Total 7.9 MB/s | 7.0 MB 00:00 2024-04-29T20:10:26.470 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:26.482 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:26.482 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:26.586 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:26.586 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-29T20:10:27.157 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-29T20:10:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-29T20:10:27.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-29T20:10:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-29T20:10:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 6/6 2024-04-29T20:10:28.615 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 6/6 2024-04-29T20:10:28.616 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/6 2024-04-29T20:10:28.616 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-29T20:10:28.616 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-29T20:10:28.616 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-29T20:10:28.616 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-29T20:10:28.804 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 675 kB/s | 6.0 MB 00:09 2024-04-29T20:10:28.805 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:28.805 INFO:teuthology.orchestra.run.smithi167.stdout:Total 764 kB/s | 7.0 MB 00:09 2024-04-29T20:10:28.878 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-29T20:10:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:28.884 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:28.885 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:28.889 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:28.890 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:28.986 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:28.986 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:29.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-04-29T20:10:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:29.453 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-29T20:10:29.505 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-29T20:10:29.642 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:40 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:29.765 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:29.766 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-3150.g354447ca.el9 ceph 891 k 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-04-29T20:10:29.767 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:29.768 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 971 k 2024-04-29T20:10:29.768 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-04-29T20:10:29.768 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 562 kB/s | 80 kB 00:00 2024-04-29T20:10:30.031 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-29T20:10:30.116 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-29T20:10:30.179 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-29T20:10:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 6/6 2024-04-29T20:10:30.674 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-3150.g354447ca.el9.x86_ 1.0 MB/s | 891 kB 00:00 2024-04-29T20:10:30.674 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:30.674 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.0 MB/s | 971 kB 00:00 2024-04-29T20:10:30.687 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:30.712 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:30.712 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:30.796 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:30.796 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 6/6 2024-04-29T20:10:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 1/6 2024-04-29T20:10:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-29T20:10:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-29T20:10:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-29T20:10:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-29T20:10:30.985 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-29T20:10:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 2/2 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-29T20:10:31.208 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-29T20:10:31.209 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:31.209 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:31.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse 2024-04-29T20:10:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 2/2 2024-04-29T20:10:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/2 2024-04-29T20:10:31.959 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:43 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:32.068 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-29T20:10:32.068 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:32.068 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:32.068 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-29T20:10:32.068 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:32.068 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:32.078 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:19.0.0-3150.g354447ca.el9 ceph 891 k 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:32.079 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-04-29T20:10:32.080 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:32.080 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 971 k 2024-04-29T20:10:32.080 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.8 M 2024-04-29T20:10:32.080 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:32.330 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 600 kB/s | 80 kB 00:00 2024-04-29T20:10:32.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-04-29T20:10:32.847 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-fuse-19.0.0-3150.g354447ca.el9.x86_ 1.3 MB/s | 891 kB 00:00 2024-04-29T20:10:32.848 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:32.848 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.2 MB/s | 971 kB 00:00 2024-04-29T20:10:32.860 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:32.871 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:32.882 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:32.882 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:32.960 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:32.960 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:32.997 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 273 k 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-04-29T20:10:32.998 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:32.999 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 350 k 2024-04-29T20:10:32.999 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-04-29T20:10:32.999 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:33.141 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:33.187 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 677 kB/s | 77 kB 00:00 2024-04-29T20:10:33.290 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-29T20:10:33.361 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 2/2 2024-04-29T20:10:33.604 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-3150.g354447ca.el9.no 514 kB/s | 273 kB 00:00 2024-04-29T20:10:33.604 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:33.604 INFO:teuthology.orchestra.run.smithi064.stdout:Total 577 kB/s | 350 kB 00:00 2024-04-29T20:10:33.617 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:33.628 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:33.628 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:33.656 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:33.656 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:33.801 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-29T20:10:34.017 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 2/2 2024-04-29T20:10:34.017 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/2 2024-04-29T20:10:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 2/2 2024-04-29T20:10:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 2/2 2024-04-29T20:10:34.048 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:10:34.048 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-29T20:10:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:34.264 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-29T20:10:34.264 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:34.264 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:34.265 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-29T20:10:34.265 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:34.265 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:34.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-volume 2024-04-29T20:10:34.740 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 1/2 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:35.032 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:35.048 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:47 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:35.171 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:35.173 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-04-29T20:10:35.173 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:35.173 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume noarch 2:19.0.0-3150.g354447ca.el9 ceph-noarch 273 k 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:35.174 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 350 k 2024-04-29T20:10:35.175 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.5 M 2024-04-29T20:10:35.175 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:35.319 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 1.0 MB/s | 77 kB 00:00 2024-04-29T20:10:35.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-04-29T20:10:35.486 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-volume-19.0.0-3150.g354447ca.el9.no 1.1 MB/s | 273 kB 00:00 2024-04-29T20:10:35.487 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:35.487 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.1 MB/s | 350 kB 00:00 2024-04-29T20:10:35.499 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:35.508 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:35.508 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:35.534 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:35.534 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:35.679 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:35.852 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-29T20:10:35.897 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 2/2 2024-04-29T20:10:35.906 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:46 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:35.912 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 2/2 2024-04-29T20:10:35.912 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:10:35.912 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-29T20:10:35.913 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:36.032 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-3150.g354447ca.el9 ceph 127 k 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-04-29T20:10:36.033 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:36.618 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 1/2 2024-04-29T20:10:36.660 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-3150.g354447ca.el9.x86_64 202 kB/s | 127 kB 00:00 2024-04-29T20:10:36.661 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:36.661 INFO:teuthology.orchestra.run.smithi064.stdout:Total 202 kB/s | 127 kB 00:00 2024-04-29T20:10:36.661 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:36.667 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:36.667 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:36.735 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:36.735 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:36.915 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-29T20:10:36.916 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:36.916 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:36.916 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume-2:19.0.0-3150.g354447ca.el9.noarch 2024-04-29T20:10:36.916 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-29T20:10:36.916 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:36.916 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:37.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel 2024-04-29T20:10:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:37.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:37.642 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:37.642 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:37.642 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:37.642 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:37.642 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:37.774 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:49 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:37.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-04-29T20:10:37.898 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:37.899 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:37.899 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:10:37.899 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:37.899 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:37.899 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:19.0.0-3150.g354447ca.el9 ceph 127 k 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 127 k 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 456 k 2024-04-29T20:10:37.900 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:38.393 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-19.0.0-3150.g354447ca.el9.x86_64 257 kB/s | 127 kB 00:00 2024-04-29T20:10:38.393 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:38.394 INFO:teuthology.orchestra.run.smithi167.stdout:Total 256 kB/s | 127 kB 00:00 2024-04-29T20:10:38.394 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:38.400 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:38.400 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:38.424 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:49 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:38.467 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:38.467 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:38.505 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:38.548 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:38.549 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:38.549 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:38.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-04-29T20:10:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:38.731 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:39.163 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:39.287 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-3150.g354447ca.el9 ceph 32 k 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:10:39.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:39.289 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 32 k 2024-04-29T20:10:39.289 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 142 k 2024-04-29T20:10:39.289 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:39.401 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:39.401 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:39.401 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:39.401 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:39.401 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:39.401 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:39.504 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-3150.g354447ca.el9.x86_6 147 kB/s | 32 kB 00:00 2024-04-29T20:10:39.504 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:39.505 INFO:teuthology.orchestra.run.smithi064.stdout:Total 146 kB/s | 32 kB 00:00 2024-04-29T20:10:39.505 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:39.509 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:39.509 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:39.511 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:39.511 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:39.602 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:39.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2 2024-04-29T20:10:39.728 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:40.071 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:40.185 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:52 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:40.267 INFO:teuthology.orchestra.run.smithi167.stdout:Package libcephfs2-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:40.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:40.281 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:40.281 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:40.281 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:40.282 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:40.282 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:40.308 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:40.309 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:40.309 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:40.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs-devel 2024-04-29T20:10:40.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-04-29T20:10:40.908 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:52 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:41.001 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:52 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:41.028 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:41.029 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:41.029 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:10:41.029 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:41.029 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:41.029 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:19.0.0-3150.g354447ca.el9 ceph 32 k 2024-04-29T20:10:41.029 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 32 k 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 142 k 2024-04-29T20:10:41.030 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:41.081 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:41.125 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:41.126 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:41.126 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:41.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-04-29T20:10:41.522 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-devel-19.0.0-3150.g354447ca.el9.x86_6 64 kB/s | 32 kB 00:00 2024-04-29T20:10:41.522 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:41.523 INFO:teuthology.orchestra.run.smithi167.stdout:Total 64 kB/s | 32 kB 00:00 2024-04-29T20:10:41.523 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:41.526 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:41.526 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:41.529 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:41.529 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:41.610 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:41.726 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:52 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:41.808 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:41.851 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:41.852 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:41.852 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:41.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-04-29T20:10:42.062 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:42.277 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:42.277 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:42.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2 2024-04-29T20:10:42.458 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:53 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:42.540 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:42.584 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:42.585 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:42.585 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:42.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-04-29T20:10:42.960 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:54 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:43.039 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:43.080 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:43.080 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:43.080 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:43.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1 2024-04-29T20:10:43.181 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:54 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:43.264 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:43.309 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:43.310 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:43.310 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:43.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-04-29T20:10:43.651 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:55 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:43.730 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:43.771 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:43.772 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:43.772 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:43.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados 2024-04-29T20:10:43.913 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:54 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:43.994 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:44.037 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:44.038 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:44.038 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:44.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-04-29T20:10:44.349 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:56 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:44.430 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rados-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:44.472 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:44.473 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:44.473 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:44.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw 2024-04-29T20:10:44.643 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:55 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:44.725 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:44.769 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:44.770 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:10:44.770 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:44.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-04-29T20:10:45.049 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:57 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:45.128 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rgw-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:45.169 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:45.170 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:45.170 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:45.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs 2024-04-29T20:10:45.494 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:56 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:45.617 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:45.618 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:45.618 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-3150.g354447ca.el9 ceph 85 k 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-04-29T20:10:45.619 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:45.757 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:57 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:45.836 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-cephfs-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:45.876 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:45.877 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:45.877 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:45.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd 2024-04-29T20:10:46.099 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-3150.g354447ca.el9.x86_64.rpm 178 kB/s | 85 kB 00:00 2024-04-29T20:10:46.099 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:46.099 INFO:teuthology.orchestra.run.smithi064.stdout:Total 177 kB/s | 85 kB 00:00 2024-04-29T20:10:46.100 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:46.107 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:46.107 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:46.176 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:46.177 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:46.436 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:46.447 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:58 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:46.527 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rbd-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:10:46.548 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:46.568 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:46.569 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:10:46.569 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:46.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse 2024-04-29T20:10:46.905 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:47.140 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:47.141 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:47.154 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:59 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:47.274 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:47.275 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:47.275 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-29T20:10:47.275 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:47.275 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:47.275 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:19.0.0-3150.g354447ca.el9 ceph 85 k 2024-04-29T20:10:47.275 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 85 k 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 231 k 2024-04-29T20:10:47.276 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:47.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-04-29T20:10:47.775 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-19.0.0-3150.g354447ca.el9.x86_64.rpm 171 kB/s | 85 kB 00:00 2024-04-29T20:10:47.775 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:47.775 INFO:teuthology.orchestra.run.smithi167.stdout:Total 170 kB/s | 85 kB 00:00 2024-04-29T20:10:47.776 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:47.783 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:47.783 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:47.847 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:47.847 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:47.921 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:58 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:47.989 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:48.046 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:48.046 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-3150.g354447ca.el9 ceph 3.1 M 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-04-29T20:10:48.047 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:48.104 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:48.441 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:48.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:48.670 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:48.670 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:48.670 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:48.670 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:48.670 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:48.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror 2024-04-29T20:10:49.435 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:01 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:49.558 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:19.0.0-3150.g354447ca.el9 ceph 3.1 M 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.1 M 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 13 M 2024-04-29T20:10:49.559 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:50.702 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-3150.g354447ca.el9.x86_64.rpm 1.2 MB/s | 3.1 MB 00:02 2024-04-29T20:10:50.703 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:50.703 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 3.1 MB 00:02 2024-04-29T20:10:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:50.716 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:50.716 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:50.805 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:50.806 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:51.253 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:51.346 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:10:51.346 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-29T20:10:51.346 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-29T20:10:51.346 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-29T20:10:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:52.431 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:52.431 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:52.432 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:52.432 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:52.432 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:52.432 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:52.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-04-29T20:10:52.654 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-19.0.0-3150.g354447ca.el9.x86_64.rpm 1.0 MB/s | 3.1 MB 00:03 2024-04-29T20:10:52.655 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:52.655 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.0 MB/s | 3.1 MB 00:03 2024-04-29T20:10:52.655 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:52.666 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:52.666 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:52.751 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:52.752 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:53.184 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:04 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:53.254 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:53.279 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:53.279 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-29T20:10:53.279 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-29T20:10:53.279 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-29T20:10:53.280 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-29T20:10:53.280 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:53.312 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-3150.g354447ca.el9 ceph 171 k 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 171 k 2024-04-29T20:10:53.313 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-04-29T20:10:53.314 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:53.755 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-3150.g354447ca.el9.x86_64.rpm 388 kB/s | 171 kB 00:00 2024-04-29T20:10:53.756 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:53.756 INFO:teuthology.orchestra.run.smithi064.stdout:Total 386 kB/s | 171 kB 00:00 2024-04-29T20:10:53.756 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:10:53.765 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:10:53.765 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:10:53.833 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:10:53.833 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:10:53.994 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:10:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:54.319 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:54.319 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:54.319 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:54.319 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:54.320 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:54.320 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:54.532 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:54.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd 2024-04-29T20:10:54.770 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:54.770 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:54.770 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:10:54.770 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:54.770 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:54.770 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:10:54.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base-debuginfo 2024-04-29T20:10:55.080 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:07 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:55.204 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:19.0.0-3150.g354447ca.el9 ceph 171 k 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:55.205 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 171 k 2024-04-29T20:10:55.206 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 494 k 2024-04-29T20:10:55.206 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:10:55.537 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:06 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:10:55.662 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 69 M 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 24 M 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-04-29T20:10:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:10:55.664 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 93 M 2024-04-29T20:10:55.664 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 589 M 2024-04-29T20:10:55.664 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:10:55.807 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-19.0.0-3150.g354447ca.el9.x86_64.rpm 285 kB/s | 171 kB 00:00 2024-04-29T20:10:55.808 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:10:55.808 INFO:teuthology.orchestra.run.smithi167.stdout:Total 284 kB/s | 171 kB 00:00 2024-04-29T20:10:55.808 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:10:55.816 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:10:55.816 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:10:55.880 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:10:55.881 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:10:56.037 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:10:56.127 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:56.786 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:10:56.786 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:56.786 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:10:56.786 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:10:56.786 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:56.786 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:10:56.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base-debuginfo 2024-04-29T20:10:57.518 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:09 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:10:57.641 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:10:57.641 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:57.641 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 69 M 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 24 M 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-04-29T20:10:57.642 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:10:57.643 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 93 M 2024-04-29T20:10:57.643 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 589 M 2024-04-29T20:10:57.643 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:11:02.169 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): ceph-debuginfo-19.0.0-3150.g354447ca.el9 5.4 MB/s | 24 MB 00:04 2024-04-29T20:11:02.591 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): ceph-debuginfo-19.0.0-3150.g354447ca.el9 3.5 MB/s | 24 MB 00:06 2024-04-29T20:11:06.566 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-base-debuginfo-19.0.0-3150.g354447c 6.3 MB/s | 69 MB 00:10 2024-04-29T20:11:06.567 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:11:06.568 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.5 MB/s | 93 MB 00:10 2024-04-29T20:11:06.568 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:11:06.573 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:11:06.573 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:11:07.188 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:11:07.188 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:11:11.268 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:11:18.888 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/2 2024-04-29T20:11:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 2/2 2024-04-29T20:11:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 2/2 2024-04-29T20:11:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/2 2024-04-29T20:11:22.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2/2 2024-04-29T20:11:22.993 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:11:22.993 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:11:22.993 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:11:22.993 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:11:22.993 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:11:22.994 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:11:23.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-common-debuginfo 2024-04-29T20:11:23.367 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-base-debuginfo-19.0.0-3150.g354447c 2.7 MB/s | 69 MB 00:25 2024-04-29T20:11:23.368 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:11:23.369 INFO:teuthology.orchestra.run.smithi167.stdout:Total 3.6 MB/s | 93 MB 00:25 2024-04-29T20:11:23.369 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:11:23.374 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:11:23.374 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:11:23.753 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:34 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:11:23.877 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 454 M 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:11:23.878 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:11:23.879 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:11:23.879 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:11:23.879 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:11:23.879 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 454 M 2024-04-29T20:11:23.879 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.0 G 2024-04-29T20:11:23.879 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:11:23.958 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:11:23.959 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:11:27.808 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:11:34.968 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/2 2024-04-29T20:11:35.051 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 2/2 2024-04-29T20:11:39.071 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 2/2 2024-04-29T20:11:39.071 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/2 2024-04-29T20:11:39.289 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2/2 2024-04-29T20:11:39.289 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:11:39.290 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:11:39.290 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:11:39.290 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:11:39.290 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:11:39.290 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:11:39.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-common-debuginfo 2024-04-29T20:11:40.034 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:52 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:11:40.153 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 454 M 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:11:40.154 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:11:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:11:40.155 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 454 M 2024-04-29T20:11:40.155 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.0 G 2024-04-29T20:11:40.155 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:12:07.519 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-common-debuginfo-19.0.0-3150.g354447ca.el9 10 MB/s | 454 MB 00:43 2024-04-29T20:12:07.519 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:12:07.519 INFO:teuthology.orchestra.run.smithi064.stdout:Total 10 MB/s | 454 MB 00:43 2024-04-29T20:12:07.520 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:12:07.524 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:12:07.524 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:12:10.350 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:12:10.350 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:12:46.441 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-common-debuginfo-19.0.0-3150.g354447ca.el9 6.9 MB/s | 454 MB 01:06 2024-04-29T20:12:46.442 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:12:46.442 INFO:teuthology.orchestra.run.smithi167.stdout:Total 6.9 MB/s | 454 MB 01:06 2024-04-29T20:12:46.442 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:12:46.447 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:12:46.447 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:12:49.136 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:12:49.136 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:13:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:13:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:13:09.128 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:13:09.356 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:13:09.356 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:09.356 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:13:09.356 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:13:09.357 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:09.357 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:13:09.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-04-29T20:13:10.244 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:21 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:13:10.370 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.5 M 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.5 M 2024-04-29T20:13:10.371 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 6.6 M 2024-04-29T20:13:10.372 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:13:11.425 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-31 1.4 MB/s | 1.5 MB 00:01 2024-04-29T20:13:11.425 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:13:11.425 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.4 MB/s | 1.5 MB 00:01 2024-04-29T20:13:11.426 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:13:11.429 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:13:11.430 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:13:11.477 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:13:11.478 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:13:11.747 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:13:11.856 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-3150. 1/1 2024-04-29T20:13:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-3150. 1/1 2024-04-29T20:13:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-3150. 1/1 2024-04-29T20:13:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:12.436 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:13:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:13:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:12.437 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:13:12.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw-debuginfo 2024-04-29T20:13:13.152 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:24 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:13:13.282 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 190 M 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 190 M 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 964 M 2024-04-29T20:13:13.283 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:13:36.157 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:13:36.317 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:13:45.956 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:13:46.170 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:13:46.171 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:46.171 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:13:46.171 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:13:46.171 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:46.171 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:13:46.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-04-29T20:13:46.915 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:04:58 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:13:47.036 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.5 M 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-04-29T20:13:47.037 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:13:47.038 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:47.038 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.5 M 2024-04-29T20:13:47.038 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 6.6 M 2024-04-29T20:13:47.038 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:13:48.290 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-31 1.2 MB/s | 1.5 MB 00:01 2024-04-29T20:13:48.290 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:13:48.290 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.2 MB/s | 1.5 MB 00:01 2024-04-29T20:13:48.291 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:13:48.295 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:13:48.295 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:13:48.342 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:13:48.342 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:13:48.612 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:13:48.711 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-3150. 1/1 2024-04-29T20:13:49.056 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-3150. 1/1 2024-04-29T20:13:49.261 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-3150. 1/1 2024-04-29T20:13:49.262 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:49.262 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:13:49.262 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:13:49.262 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:49.262 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:13:49.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw-debuginfo 2024-04-29T20:13:49.931 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:01 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:13:50.054 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 190 M 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:13:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 190 M 2024-04-29T20:13:50.056 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 964 M 2024-04-29T20:13:50.056 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:14:10.637 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-radosgw-debuginfo-19.0.0-3150.g354447ca.el 3.3 MB/s | 190 MB 00:57 2024-04-29T20:14:10.638 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:14:10.638 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.3 MB/s | 190 MB 00:57 2024-04-29T20:14:10.638 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:14:10.643 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:14:10.643 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:14:11.846 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:14:11.846 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:14:23.253 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-radosgw-debuginfo-19.0.0-3150.g354447ca.el 5.7 MB/s | 190 MB 00:33 2024-04-29T20:14:23.254 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:14:23.254 INFO:teuthology.orchestra.run.smithi167.stdout:Total 5.7 MB/s | 190 MB 00:33 2024-04-29T20:14:23.255 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:14:23.259 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:14:23.259 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:14:24.406 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:14:24.406 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:14:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:14:33.693 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x 1/1 2024-04-29T20:14:39.339 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x 1/1 2024-04-29T20:14:39.570 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x 1/1 2024-04-29T20:14:39.570 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:14:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:14:39.570 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:14:39.570 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:14:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:14:39.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test-debuginfo 2024-04-29T20:14:40.300 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:51 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:14:40.425 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 703 M 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:14:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:14:40.427 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 703 M 2024-04-29T20:14:40.427 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 3.6 G 2024-04-29T20:14:40.427 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:14:45.160 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:14:45.337 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x 1/1 2024-04-29T20:14:51.338 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x 1/1 2024-04-29T20:14:51.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x 1/1 2024-04-29T20:14:51.552 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:14:51.552 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:14:51.552 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:14:51.552 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:14:51.552 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:14:51.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test-debuginfo 2024-04-29T20:14:52.268 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:04 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:14:52.389 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 703 M 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:14:52.390 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:14:52.394 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 703 M 2024-04-29T20:14:52.394 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 3.6 G 2024-04-29T20:14:52.394 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:16:05.179 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-test-debuginfo-19.0.0-3150.g354447ca.el9.x 8.3 MB/s | 703 MB 01:24 2024-04-29T20:16:05.179 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:16:05.180 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.3 MB/s | 703 MB 01:24 2024-04-29T20:16:05.180 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:16:05.186 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:16:05.186 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:16:09.536 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:16:09.536 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:16:28.127 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-test-debuginfo-19.0.0-3150.g354447ca.el9.x 7.3 MB/s | 703 MB 01:35 2024-04-29T20:16:28.127 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:16:28.128 INFO:teuthology.orchestra.run.smithi167.stdout:Total 7.3 MB/s | 703 MB 01:35 2024-04-29T20:16:28.128 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:16:28.134 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:16:28.134 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:16:32.310 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:16:32.311 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:17:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:17:31.165 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:17:41.026 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:17:41.242 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:17:41.242 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:41.242 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:17:41.242 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:17:41.242 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:41.243 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:17:41.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base-debuginfo 2024-04-29T20:17:42.081 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:53 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:17:42.163 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:17:42.210 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:17:42.211 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-29T20:17:42.211 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:17:42.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-debuginfo 2024-04-29T20:17:42.808 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:53 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:17:42.933 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 20 M 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:17:42.934 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:17:42.935 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:17:42.935 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:42.935 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 20 M 2024-04-29T20:17:42.935 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 104 M 2024-04-29T20:17:42.935 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:17:47.411 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mgr-debuginfo-19.0.0-3150.g354447ca.el9.x8 4.5 MB/s | 20 MB 00:04 2024-04-29T20:17:47.414 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:17:47.414 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.5 MB/s | 20 MB 00:04 2024-04-29T20:17:47.414 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:17:47.416 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:17:47.417 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:17:47.579 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:17:47.579 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:17:49.774 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:17:49.879 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:17:49.985 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:17:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:17:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:17:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:17:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:51.233 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:17:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:17:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:51.233 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:17:51.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mds-debuginfo 2024-04-29T20:17:51.984 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:09:02 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:17:52.110 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:17:52.110 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:17:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 54 M 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:17:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:17:52.112 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 54 M 2024-04-29T20:17:52.112 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 196 M 2024-04-29T20:17:52.112 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:17:54.938 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mds-debuginfo-19.0.0-3150.g354447ca.el9.x8 19 MB/s | 54 MB 00:02 2024-04-29T20:17:54.939 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:17:54.939 INFO:teuthology.orchestra.run.smithi064.stdout:Total 19 MB/s | 54 MB 00:02 2024-04-29T20:17:54.939 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:17:54.943 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:17:54.944 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:17:55.311 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:17:55.311 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:18:00.262 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:18:00.492 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:18:00.493 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:00.493 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:18:00.493 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:18:00.493 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:00.493 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:18:00.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base-debuginfo 2024-04-29T20:18:00.972 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:18:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:01.298 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:09:13 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:18:01.377 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 is already installed. 2024-04-29T20:18:01.420 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:18:01.420 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-04-29T20:18:01.420 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:18:01.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-debuginfo 2024-04-29T20:18:01.997 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:09:13 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:18:02.116 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:18:02.117 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:02.117 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:18:02.117 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:02.117 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:18:02.117 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 20 M 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 20 M 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 104 M 2024-04-29T20:18:02.118 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:18:02.520 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:02.780 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:02.781 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:02.781 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:18:02.781 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:18:02.781 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:02.781 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:18:02.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mon-debuginfo 2024-04-29T20:18:03.502 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:09:14 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:18:03.629 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 99 M 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 99 M 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 451 M 2024-04-29T20:18:03.630 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:18:05.444 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mgr-debuginfo-19.0.0-3150.g354447ca.el9.x8 6.1 MB/s | 20 MB 00:03 2024-04-29T20:18:05.445 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:18:05.445 INFO:teuthology.orchestra.run.smithi167.stdout:Total 6.0 MB/s | 20 MB 00:03 2024-04-29T20:18:05.445 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:18:05.449 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:18:05.449 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:18:05.605 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:18:05.605 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:18:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:18:08.014 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:08.953 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:09.157 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:18:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:18:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:09.157 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:18:09.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mds-debuginfo 2024-04-29T20:18:09.849 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:09:21 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 54 M 2024-04-29T20:18:09.969 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 54 M 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 196 M 2024-04-29T20:18:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:18:12.389 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mon-debuginfo-19.0.0-3150.g354447ca.el9.x8 11 MB/s | 99 MB 00:08 2024-04-29T20:18:12.390 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:18:12.391 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 99 MB 00:08 2024-04-29T20:18:12.391 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:18:12.396 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:18:12.396 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:18:13.041 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:18:13.042 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:18:16.188 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mds-debuginfo-19.0.0-3150.g354447ca.el9.x8 8.6 MB/s | 54 MB 00:06 2024-04-29T20:18:16.189 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:18:16.189 INFO:teuthology.orchestra.run.smithi167.stdout:Total 8.6 MB/s | 54 MB 00:06 2024-04-29T20:18:16.189 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:18:16.193 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:18:16.194 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:18:16.550 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:18:16.550 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:18:21.929 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:18:22.033 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:23.522 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:23.737 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:23.738 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:23.738 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:18:23.738 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:18:23.738 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:23.738 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:18:23.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mon-debuginfo 2024-04-29T20:18:23.987 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:18:24.089 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:24.442 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:09:36 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:18:24.563 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 99 M 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:18:24.564 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:24.565 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 99 M 2024-04-29T20:18:24.565 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 451 M 2024-04-29T20:18:24.565 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:18:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:27.134 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:18:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:18:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:27.134 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:18:27.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-osd-debuginfo 2024-04-29T20:18:27.858 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:09:38 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:18:27.984 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:18:27.984 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 290 M 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 290 M 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.2 G 2024-04-29T20:18:27.985 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:18:34.240 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mon-debuginfo-19.0.0-3150.g354447ca.el9.x8 10 MB/s | 99 MB 00:09 2024-04-29T20:18:34.241 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:18:34.241 INFO:teuthology.orchestra.run.smithi167.stdout:Total 10 MB/s | 99 MB 00:09 2024-04-29T20:18:34.242 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:18:34.245 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:18:34.246 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:18:34.860 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:18:34.860 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:18:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:18:45.585 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:48.590 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:18:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:48.811 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:18:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:18:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:48.812 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:18:49.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-osd-debuginfo 2024-04-29T20:18:49.524 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:10:01 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:18:49.648 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 290 M 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:18:49.649 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:18:49.650 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:18:49.650 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 290 M 2024-04-29T20:18:49.650 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.2 G 2024-04-29T20:18:49.650 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:18:55.155 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-osd-debuginfo-19.0.0-3150.g354447ca.el9.x8 11 MB/s | 290 MB 00:27 2024-04-29T20:18:55.156 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:18:55.157 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 290 MB 00:27 2024-04-29T20:18:55.157 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:18:55.161 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:18:55.161 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:18:56.975 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:18:56.975 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:19:13.973 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-osd-debuginfo-19.0.0-3150.g354447ca.el9.x8 12 MB/s | 290 MB 00:24 2024-04-29T20:19:13.974 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:13.974 INFO:teuthology.orchestra.run.smithi167.stdout:Total 12 MB/s | 290 MB 00:24 2024-04-29T20:19:13.975 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:19:13.979 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:19:13.979 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:19:15.712 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:19:16.147 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:19:28.345 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:19:28.468 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:19:35.747 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:19:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:19:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:36.041 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:19:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:36.041 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:19:36.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse-debuginfo 2024-04-29T20:19:36.788 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:10:47 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:19:36.912 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 6.1 M 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:19:36.913 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:36.914 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:19:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:36.914 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.1 M 2024-04-29T20:19:36.914 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 29 M 2024-04-29T20:19:36.914 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:19:38.789 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-fuse-debuginfo-19.0.0-3150.g354447ca.el9.x 3.2 MB/s | 6.1 MB 00:01 2024-04-29T20:19:38.789 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:38.790 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.2 MB/s | 6.1 MB 00:01 2024-04-29T20:19:38.790 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:19:38.794 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:19:38.794 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:19:38.874 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:19:38.874 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:19:39.672 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:19:39.778 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:40.640 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:40.640 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:40.641 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:19:40.641 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:40.641 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:40.641 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:19:40.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel-debuginfo 2024-04-29T20:19:41.337 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:10:52 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:19:41.462 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 109 k 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:41.463 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:19:41.464 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:41.464 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 109 k 2024-04-29T20:19:41.464 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 440 k 2024-04-29T20:19:41.464 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:19:42.075 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-debuginfo-19.0.0-3150.g354447ca. 178 kB/s | 109 kB 00:00 2024-04-29T20:19:42.075 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:42.076 INFO:teuthology.orchestra.run.smithi064.stdout:Total 178 kB/s | 109 kB 00:00 2024-04-29T20:19:42.076 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:19:42.079 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:19:42.079 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:19:42.118 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:19:42.119 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:19:42.247 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:19:42.348 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:19:42.693 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:19:42.908 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:19:42.908 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:42.908 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:19:42.908 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:42.909 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:42.909 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:19:43.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2-debuginfo 2024-04-29T20:19:43.627 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:10:54 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:19:43.755 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 5.5 M 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:19:43.756 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:43.757 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 5.5 M 2024-04-29T20:19:43.757 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 25 M 2024-04-29T20:19:43.757 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:19:44.453 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs2-debuginfo-19.0.0-3150.g354447ca.el9. 7.8 MB/s | 5.5 MB 00:00 2024-04-29T20:19:44.453 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:44.454 INFO:teuthology.orchestra.run.smithi064.stdout:Total 7.8 MB/s | 5.5 MB 00:00 2024-04-29T20:19:44.454 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:19:44.458 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:19:44.458 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:19:44.529 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:19:44.529 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:19:45.245 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:19:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:19:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:19:45.525 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:19:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:19:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:19:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:46.108 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:19:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:46.108 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:19:46.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2-debuginfo 2024-04-29T20:19:46.847 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:10:57 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:19:46.972 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 45 M 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:19:46.973 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:46.974 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 45 M 2024-04-29T20:19:46.974 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 257 M 2024-04-29T20:19:46.974 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:19:49.389 INFO:teuthology.orchestra.run.smithi064.stdout:librados2-debuginfo-19.0.0-3150.g354447ca.el9.x 19 MB/s | 45 MB 00:02 2024-04-29T20:19:49.390 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:49.390 INFO:teuthology.orchestra.run.smithi064.stdout:Total 19 MB/s | 45 MB 00:02 2024-04-29T20:19:49.390 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:19:49.394 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:19:49.395 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:19:49.714 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:19:49.715 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:19:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:19:53.575 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:19:53.575 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:53.575 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:19:53.575 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:53.575 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:53.575 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:19:53.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse-debuginfo 2024-04-29T20:19:54.301 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:06 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:19:54.428 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 6.1 M 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:54.429 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:19:54.430 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:54.430 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:19:54.430 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:54.430 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 6.1 M 2024-04-29T20:19:54.430 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 29 M 2024-04-29T20:19:54.430 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:19:55.181 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:19:55.265 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-fuse-debuginfo-19.0.0-3150.g354447ca.el9.x 7.3 MB/s | 6.1 MB 00:00 2024-04-29T20:19:55.265 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:55.266 INFO:teuthology.orchestra.run.smithi167.stdout:Total 7.3 MB/s | 6.1 MB 00:00 2024-04-29T20:19:55.266 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:19:55.270 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:19:55.270 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:19:55.279 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:55.343 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:19:55.343 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:19:56.107 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:19:56.203 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:56.748 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:56.965 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:56.966 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:56.966 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:19:56.966 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:56.966 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:56.966 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:19:57.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel-debuginfo 2024-04-29T20:19:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:19:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:57.528 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:19:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:57.528 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:19:57.651 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:09 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:19:57.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1-debuginfo 2024-04-29T20:19:57.775 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 109 k 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:19:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:57.777 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:19:57.777 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:57.777 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 109 k 2024-04-29T20:19:57.777 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 440 k 2024-04-29T20:19:57.777 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:19:58.252 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-debuginfo-19.0.0-3150.g354447ca. 229 kB/s | 109 kB 00:00 2024-04-29T20:19:58.252 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:19:58.253 INFO:teuthology.orchestra.run.smithi167.stdout:Total 228 kB/s | 109 kB 00:00 2024-04-29T20:19:58.253 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:19:58.256 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:19:58.256 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:19:58.262 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:09 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:19:58.292 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:19:58.293 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:19:58.389 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:19:58.389 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:58.389 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 52 M 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 52 M 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 309 M 2024-04-29T20:19:58.390 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:19:58.429 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:19:58.539 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:19:58.866 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:19:59.073 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:19:59.074 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:59.074 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:19:59.074 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:19:59.074 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:59.074 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:19:59.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2-debuginfo 2024-04-29T20:19:59.767 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:11 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:19:59.887 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:19:59.890 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:59.890 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:19:59.890 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 5.5 M 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:19:59.891 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 5.5 M 2024-04-29T20:19:59.892 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 25 M 2024-04-29T20:19:59.892 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:01.042 INFO:teuthology.orchestra.run.smithi064.stdout:librbd1-debuginfo-19.0.0-3150.g354447ca.el9.x86 20 MB/s | 52 MB 00:02 2024-04-29T20:20:01.043 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:01.043 INFO:teuthology.orchestra.run.smithi064.stdout:Total 20 MB/s | 52 MB 00:02 2024-04-29T20:20:01.043 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:01.048 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:01.048 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:01.411 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:01.412 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:07.804 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:07.891 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:08.446 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs2-debuginfo-19.0.0-3150.g354447ca.el9. 653 kB/s | 5.5 MB 00:08 2024-04-29T20:20:08.447 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:08.447 INFO:teuthology.orchestra.run.smithi167.stdout:Total 653 kB/s | 5.5 MB 00:08 2024-04-29T20:20:08.447 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:08.451 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:08.451 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:08.521 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:08.521 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:20:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:20:10.024 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:10.057 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:20:10.057 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:10.057 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:10.057 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:10.057 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:10.057 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:10.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2-debuginfo 2024-04-29T20:20:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:10.383 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:10.384 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:10.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs-debuginfo 2024-04-29T20:20:10.784 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:22 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:10.904 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 45 M 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:10.905 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 45 M 2024-04-29T20:20:10.906 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 257 M 2024-04-29T20:20:10.906 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:11.136 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:22 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:11.262 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:11.263 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:11.263 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:11.263 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:11.263 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:11.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 615 k 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 615 k 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.1 M 2024-04-29T20:20:11.264 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:12.453 INFO:teuthology.orchestra.run.smithi064.stdout:python3-cephfs-debuginfo-19.0.0-3150.g354447ca. 517 kB/s | 615 kB 00:01 2024-04-29T20:20:12.453 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:12.454 INFO:teuthology.orchestra.run.smithi064.stdout:Total 516 kB/s | 615 kB 00:01 2024-04-29T20:20:12.454 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:12.458 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:12.458 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:12.505 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:12.505 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:20:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:20:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:20:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:13.374 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:13.375 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:13.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados-debuginfo 2024-04-29T20:20:14.109 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:25 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:14.236 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:14.237 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:14.237 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:14.237 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 769 k 2024-04-29T20:20:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 769 k 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.7 M 2024-04-29T20:20:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:15.331 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rados-debuginfo-19.0.0-3150.g354447ca.e 704 kB/s | 769 kB 00:01 2024-04-29T20:20:15.331 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:15.332 INFO:teuthology.orchestra.run.smithi064.stdout:Total 703 kB/s | 769 kB 00:01 2024-04-29T20:20:15.332 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:15.336 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:15.336 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:15.383 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:15.383 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9. 1/1 2024-04-29T20:20:16.001 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9. 1/1 2024-04-29T20:20:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9. 1/1 2024-04-29T20:20:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:16.233 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:16.233 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:16.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd-debuginfo 2024-04-29T20:20:16.950 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:27 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:17.076 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.1 M 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:17.077 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:17.078 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.1 M 2024-04-29T20:20:17.078 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 4.3 M 2024-04-29T20:20:17.078 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:17.393 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rbd-debuginfo-19.0.0-3150.g354447ca.el9 3.6 MB/s | 1.1 MB 00:00 2024-04-29T20:20:17.393 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:17.393 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.6 MB/s | 1.1 MB 00:00 2024-04-29T20:20:17.393 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:17.397 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:17.397 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:17.442 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:17.442 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:17.669 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:18.139 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:18.375 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:18.375 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:18.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw-debuginfo 2024-04-29T20:20:19.071 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:30 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:19.197 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:19.198 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:19.198 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:19.198 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 337 k 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 337 k 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.0 M 2024-04-29T20:20:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:19.249 INFO:teuthology.orchestra.run.smithi167.stdout:librados2-debuginfo-19.0.0-3150.g354447ca.el9.x 5.4 MB/s | 45 MB 00:08 2024-04-29T20:20:19.250 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:19.250 INFO:teuthology.orchestra.run.smithi167.stdout:Total 5.4 MB/s | 45 MB 00:08 2024-04-29T20:20:19.250 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:19.254 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:19.254 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:19.558 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:19.558 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:19.578 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rgw-debuginfo-19.0.0-3150.g354447ca.el9 889 kB/s | 337 kB 00:00 2024-04-29T20:20:19.579 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:19.580 INFO:teuthology.orchestra.run.smithi064.stdout:Total 886 kB/s | 337 kB 00:00 2024-04-29T20:20:19.580 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:19.583 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:19.583 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:19.623 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:19.624 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:20.424 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:20.424 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:20.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse-debuginfo 2024-04-29T20:20:21.156 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:32 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:21.283 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:21.283 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:21.283 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:21.283 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 517 k 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 517 k 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.6 M 2024-04-29T20:20:21.284 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:21.540 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-debuginfo-19.0.0-3150.g354447ca.el9.x8 2.0 MB/s | 517 kB 00:00 2024-04-29T20:20:21.541 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:21.541 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.0 MB/s | 517 kB 00:00 2024-04-29T20:20:21.541 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:21.544 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:21.545 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:21.586 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:21.586 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:21.750 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:21.856 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:20:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:20:22.456 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:20:22.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:22.456 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:22.456 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:22.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:22.457 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:22.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror-debuginfo 2024-04-29T20:20:23.181 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:34 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:23.307 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 52 M 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 52 M 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 299 M 2024-04-29T20:20:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:24.804 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:20:26.676 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-debuginfo-19.0.0-3150.g354447ca.el9. 15 MB/s | 52 MB 00:03 2024-04-29T20:20:26.677 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:26.678 INFO:teuthology.orchestra.run.smithi064.stdout:Total 15 MB/s | 52 MB 00:03 2024-04-29T20:20:26.678 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:26.682 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:26.682 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:26.713 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:20:26.944 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_ 1/1 2024-04-29T20:20:26.944 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:26.944 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:26.944 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:26.944 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:26.945 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:27.039 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:27.039 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:27.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1-debuginfo 2024-04-29T20:20:27.637 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:39 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:27.757 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 52 M 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:27.758 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:27.759 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:27.759 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:27.759 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 52 M 2024-04-29T20:20:27.759 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 309 M 2024-04-29T20:20:27.759 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:30.627 INFO:teuthology.orchestra.run.smithi167.stdout:librbd1-debuginfo-19.0.0-3150.g354447ca.el9.x86 18 MB/s | 52 MB 00:02 2024-04-29T20:20:30.628 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:30.628 INFO:teuthology.orchestra.run.smithi167.stdout:Total 18 MB/s | 52 MB 00:02 2024-04-29T20:20:30.628 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:30.632 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:30.633 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:30.977 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:30.977 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:33.441 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:20:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:20:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:20:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:35.688 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:35.688 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:35.689 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:35.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd-debuginfo 2024-04-29T20:20:36.401 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:11:47 ago on Mon 29 Apr 2024 08:08:49 PM UTC. 2024-04-29T20:20:36.526 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-29T20:20:36.526 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:36.526 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.3 M 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.3 M 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 5.7 M 2024-04-29T20:20:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-29T20:20:37.003 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-debuginfo-19.0.0-3150.g354447ca.el9.x86 2.7 MB/s | 1.3 MB 00:00 2024-04-29T20:20:37.004 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:37.004 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-04-29T20:20:37.004 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-29T20:20:37.008 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-29T20:20:37.008 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-29T20:20:37.044 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:37.052 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-29T20:20:37.053 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-29T20:20:37.148 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-29T20:20:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:37.755 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:37.988 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-29T20:20:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:20:37.988 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-29T20:20:38.161 DEBUG:teuthology.parallel:result is None 2024-04-29T20:20:39.360 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:20:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:39.567 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:39.567 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:39.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs-debuginfo 2024-04-29T20:20:40.286 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:52 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:40.407 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 615 k 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:40.408 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 615 k 2024-04-29T20:20:40.409 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.1 M 2024-04-29T20:20:40.409 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:40.705 INFO:teuthology.orchestra.run.smithi167.stdout:python3-cephfs-debuginfo-19.0.0-3150.g354447ca. 2.0 MB/s | 615 kB 00:00 2024-04-29T20:20:40.706 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:40.707 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.0 MB/s | 615 kB 00:00 2024-04-29T20:20:40.707 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:40.709 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:40.709 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:40.747 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:40.748 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:40.913 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:41.010 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:20:41.369 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:20:41.584 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9 1/1 2024-04-29T20:20:41.585 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:41.585 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:41.585 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:41.585 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:41.585 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:41.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados-debuginfo 2024-04-29T20:20:42.278 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:54 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:42.398 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 769 k 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:42.399 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:42.400 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:42.400 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 769 k 2024-04-29T20:20:42.400 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.7 M 2024-04-29T20:20:42.400 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:42.842 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rados-debuginfo-19.0.0-3150.g354447ca.e 1.7 MB/s | 769 kB 00:00 2024-04-29T20:20:42.842 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:42.842 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.7 MB/s | 769 kB 00:00 2024-04-29T20:20:42.843 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:42.846 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:42.846 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:42.885 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:42.885 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:43.062 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:43.146 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9. 1/1 2024-04-29T20:20:43.464 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9. 1/1 2024-04-29T20:20:43.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9. 1/1 2024-04-29T20:20:43.683 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:43.683 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:43.683 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:43.684 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:43.684 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:43.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd-debuginfo 2024-04-29T20:20:44.393 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:56 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:44.513 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:44.513 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:44.513 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:44.513 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:44.513 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.1 M 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.1 M 2024-04-29T20:20:44.514 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 4.3 M 2024-04-29T20:20:44.515 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:45.037 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rbd-debuginfo-19.0.0-3150.g354447ca.el9 2.2 MB/s | 1.1 MB 00:00 2024-04-29T20:20:45.037 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:45.038 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.1 MB/s | 1.1 MB 00:00 2024-04-29T20:20:45.038 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:45.041 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:45.041 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:45.082 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:45.083 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:45.299 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:45.787 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:46.001 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:46.001 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:46.001 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:46.001 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:46.001 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:46.002 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:46.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw-debuginfo 2024-04-29T20:20:46.700 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:58 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:46.821 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:46.821 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 337 k 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 337 k 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.0 M 2024-04-29T20:20:46.822 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:47.331 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rgw-debuginfo-19.0.0-3150.g354447ca.el9 663 kB/s | 337 kB 00:00 2024-04-29T20:20:47.332 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:47.332 INFO:teuthology.orchestra.run.smithi167.stdout:Total 661 kB/s | 337 kB 00:00 2024-04-29T20:20:47.332 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:47.336 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:47.336 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:47.372 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:47.372 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:47.511 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:47.603 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:47.938 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:48.166 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x8 1/1 2024-04-29T20:20:48.166 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:48.166 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:48.166 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:48.166 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:48.166 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:48.356 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse-debuginfo 2024-04-29T20:20:48.871 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:12:00 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:49.517 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:49.517 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:49.517 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:49.517 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:49.517 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:49.517 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 517 k 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 517 k 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.6 M 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-debuginfo-19.0.0-3150.g354447ca.el9.x8 1.8 MB/s | 517 kB 00:00 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.8 MB/s | 517 kB 00:00 2024-04-29T20:20:49.518 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:49.519 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:49.519 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:49.519 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:49.519 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:20:49.519 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:20:49.565 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:20:49.920 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:20:50.133 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_6 1/1 2024-04-29T20:20:50.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:50.133 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:20:50.133 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:20:50.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:50.134 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:20:50.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror-debuginfo 2024-04-29T20:20:50.836 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:12:02 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:20:50.957 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 52 M 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:50.958 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:20:50.959 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:20:50.959 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:20:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:20:50.959 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 52 M 2024-04-29T20:20:50.959 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 299 M 2024-04-29T20:20:50.959 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:20:54.230 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-debuginfo-19.0.0-3150.g354447ca.el9. 16 MB/s | 52 MB 00:03 2024-04-29T20:20:54.231 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:20:54.231 INFO:teuthology.orchestra.run.smithi167.stdout:Total 16 MB/s | 52 MB 00:03 2024-04-29T20:20:54.231 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:20:54.235 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:20:54.236 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:20:54.572 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:20:54.573 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:21:00.514 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:21:00.609 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:21:02.818 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:21:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86 1/1 2024-04-29T20:21:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:03.047 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:21:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:21:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:03.048 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:21:03.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd-debuginfo 2024-04-29T20:21:03.741 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:12:15 ago on Mon 29 Apr 2024 08:08:48 PM UTC. 2024-04-29T20:21:03.863 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-04-29T20:21:03.863 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-3150.g354447ca.el9 ceph 1.3 M 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.3 M 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 5.7 M 2024-04-29T20:21:03.864 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-04-29T20:21:04.316 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-debuginfo-19.0.0-3150.g354447ca.el9.x86 2.9 MB/s | 1.3 MB 00:00 2024-04-29T20:21:04.316 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-04-29T20:21:04.317 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.8 MB/s | 1.3 MB 00:00 2024-04-29T20:21:04.317 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-04-29T20:21:04.320 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-04-29T20:21:04.320 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-04-29T20:21:04.363 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-04-29T20:21:04.363 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-04-29T20:21:04.603 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-04-29T20:21:04.704 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:21:05.084 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:21:05.298 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 1/1 2024-04-29T20:21:05.299 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:05.299 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-04-29T20:21:05.299 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-debuginfo-2:19.0.0-3150.g354447ca.el9.x86_64 2024-04-29T20:21:05.299 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:05.299 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-04-29T20:21:05.477 DEBUG:teuthology.parallel:result is None 2024-04-29T20:21:05.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=354447ca5357e926795d009be846687f04556beb 2024-04-29T20:21:05.619 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-29T20:21:05.645 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-3150.g354447ca.el9 2024-04-29T20:21:05.645 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3150.g354447ca.el9 2024-04-29T20:21:05.645 INFO:teuthology.task.install:The correct ceph version 19.0.0-3150.g354447ca is installed. 2024-04-29T20:21:05.647 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=354447ca5357e926795d009be846687f04556beb 2024-04-29T20:21:05.864 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-29T20:21:05.886 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-3150.g354447ca.el9 2024-04-29T20:21:05.886 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3150.g354447ca.el9 2024-04-29T20:21:05.886 INFO:teuthology.task.install:The correct ceph version 19.0.0-3150.g354447ca is installed. 2024-04-29T20:21:05.888 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-29T20:21:05.888 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:05.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-29T20:21:05.918 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:05.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-29T20:21:05.950 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-29T20:21:05.950 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:05.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-04-29T20:21:05.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-29T20:21:06.057 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:06.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-04-29T20:21:06.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-29T20:21:06.146 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-29T20:21:06.146 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:06.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-29T20:21:06.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-29T20:21:06.245 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:06.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-29T20:21:06.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-29T20:21:06.333 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-29T20:21:06.334 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:06.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-04-29T20:21:06.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-29T20:21:06.427 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:06.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-04-29T20:21:06.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-29T20:21:06.516 INFO:teuthology.run_tasks:Running task ceph... 2024-04-29T20:21:06.631 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-29T20:21:06.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-04-29T20:21:06.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2024-04-29T20:21:06.658 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-29T20:21:06.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-29T20:21:06.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-29T20:21:06.725 INFO:tasks.ceph:Creating extra log directories... 2024-04-29T20:21:06.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-29T20:21:06.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-29T20:21:06.801 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-29T20:21:06.801 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-29T20:21:06.801 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692', 'branch': 'wip-yuri6-testing-2024-04-02-1310', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7678692', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2024-04-02-1310'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '354447ca5357e926795d009be846687f04556beb', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '354447ca5357e926795d009be846687f04556beb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2024-04-02-1310', 'sha1': '354447ca5357e926795d009be846687f04556beb'}}, 'owner': 'scheduled_lflores@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1434, 'sha1': '354447ca5357e926795d009be846687f04556beb', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2024-04-02-1310', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '354447ca5357e926795d009be846687f04556beb', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNd2mtGOrCmynAlbVYW38ivB5bNl5QYQUD0ObRY1s05WIO5wjEpeyExsQSztEFoky8ntBJ1vZjctQHDJnc1VKaE=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKi49WANi6PnJX4+APe3vVQt93GlNunHK+tM60aoulD8adVf1+KW+zXdQtOTHwjo77GVRci9+1MwUWCFqAxtVwk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-29_19:49:34', 'tube': 'smithi', 'user': 'lflores', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-29T20:21:06.802 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-29T20:21:06.833 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-29T20:21:06.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-04-29T20:21:06.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2024-04-29T20:21:06.913 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:06.913 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-04-29T20:21:06.969 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-29T20:21:06.970 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-04-29T20:21:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-29 20:10:51.598352203 +0000 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-29 20:08:06.585061348 +0000 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-29 20:08:06.585061348 +0000 2024-04-29T20:21:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-29 20:08:06.585061348 +0000 2024-04-29T20:21:07.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-29T20:21:07.089 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-29T20:21:07.090 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-29T20:21:07.090 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000244795 s, 2.1 MB/s 2024-04-29T20:21:07.091 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-29T20:21:07.147 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-04-29T20:21:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-29T20:21:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:07.203 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-29T20:21:07.204 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:07.204 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:07.204 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-29 20:10:51.598352203 +0000 2024-04-29T20:21:07.204 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-29 20:08:06.930054017 +0000 2024-04-29T20:21:07.204 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-29 20:08:06.930054017 +0000 2024-04-29T20:21:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-29 20:08:06.930054017 +0000 2024-04-29T20:21:07.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-29T20:21:07.268 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-29T20:21:07.268 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-29T20:21:07.268 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000186688 s, 2.7 MB/s 2024-04-29T20:21:07.269 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-29T20:21:07.325 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-04-29T20:21:07.380 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-29T20:21:07.380 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:07.380 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-29T20:21:07.380 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:07.380 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:07.381 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-29 20:10:51.599352184 +0000 2024-04-29T20:21:07.381 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-29 20:08:07.290046367 +0000 2024-04-29T20:21:07.381 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-29 20:08:07.290046367 +0000 2024-04-29T20:21:07.381 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-29 20:08:07.290046367 +0000 2024-04-29T20:21:07.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-29T20:21:07.443 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-29T20:21:07.443 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-29T20:21:07.443 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000192171 s, 2.7 MB/s 2024-04-29T20:21:07.444 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-29T20:21:07.501 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-29 20:10:51.599352184 +0000 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-29 20:08:07.637038994 +0000 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-29 20:08:07.637038994 +0000 2024-04-29T20:21:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-29 20:08:07.637038994 +0000 2024-04-29T20:21:07.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-29T20:21:07.620 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-29T20:21:07.620 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-29T20:21:07.620 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000207288 s, 2.5 MB/s 2024-04-29T20:21:07.621 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-29T20:21:07.677 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:07.677 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:07.677 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-04-29T20:21:07.693 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-29T20:21:07.694 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-04-29T20:21:07.749 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-29T20:21:07.749 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:07.749 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-29T20:21:07.749 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:07.750 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:07.750 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-29 20:10:53.503455550 +0000 2024-04-29T20:21:07.750 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-29 20:08:04.128472613 +0000 2024-04-29T20:21:07.750 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-29 20:08:04.128472613 +0000 2024-04-29T20:21:07.750 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-29 20:08:04.128472613 +0000 2024-04-29T20:21:07.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-29T20:21:07.813 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-29T20:21:07.813 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-29T20:21:07.813 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000188192 s, 2.7 MB/s 2024-04-29T20:21:07.814 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-29T20:21:07.871 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-04-29T20:21:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-29T20:21:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:07.927 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-29T20:21:07.927 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:07.927 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:07.928 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-29 20:10:53.503455550 +0000 2024-04-29T20:21:07.928 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-29 20:08:04.469465587 +0000 2024-04-29T20:21:07.928 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-29 20:08:04.469465587 +0000 2024-04-29T20:21:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-29 20:08:04.469465587 +0000 2024-04-29T20:21:07.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-29T20:21:07.989 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-29T20:21:07.989 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-29T20:21:07.989 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.00017601 s, 2.9 MB/s 2024-04-29T20:21:07.990 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-29T20:21:08.046 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-29 20:10:53.504455532 +0000 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-29 20:08:04.809458582 +0000 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-29 20:08:04.809458582 +0000 2024-04-29T20:21:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-29 20:08:04.809458582 +0000 2024-04-29T20:21:08.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-29T20:21:08.163 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-29T20:21:08.163 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-29T20:21:08.163 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000162989 s, 3.1 MB/s 2024-04-29T20:21:08.164 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-29T20:21:08.219 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-04-29T20:21:08.273 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-29 20:10:53.504455532 +0000 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-29 20:08:05.137451825 +0000 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-29 20:08:05.137451825 +0000 2024-04-29T20:21:08.274 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-04-29 20:08:05.137451825 +0000 2024-04-29T20:21:08.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-29T20:21:08.336 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-29T20:21:08.336 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-29T20:21:08.336 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000247574 s, 2.1 MB/s 2024-04-29T20:21:08.337 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-29T20:21:08.392 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:08.392 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-29T20:21:08.392 INFO:tasks.ceph:Generating config... 2024-04-29T20:21:08.394 INFO:tasks.ceph:[client] debug monc = 20 2024-04-29T20:21:08.394 INFO:tasks.ceph:[client] debug ms = 1 2024-04-29T20:21:08.394 INFO:tasks.ceph:[client] debug objecter = 20 2024-04-29T20:21:08.394 INFO:tasks.ceph:[client] debug rados = 20 2024-04-29T20:21:08.394 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-29T20:21:08.394 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-29T20:21:08.394 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-29T20:21:08.394 INFO:tasks.ceph:[global] ms type = async 2024-04-29T20:21:08.394 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-04-29T20:21:08.394 INFO:tasks.ceph:[global] osd op thread timeout = 150 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-04-29T20:21:08.395 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-29T20:21:08.395 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-29T20:21:08.396 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd class default list = * 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd class load list = * 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-29T20:21:08.397 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-29T20:21:08.398 INFO:tasks.ceph:Setting up mon.a... 2024-04-29T20:21:08.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-29T20:21:08.447 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-04-29T20:21:08.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-29T20:21:08.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-29T20:21:08.558 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.64'), ('mon.c', '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]'), ('mon.b', '172.21.15.167')] 2024-04-29T20:21:08.559 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-29T20:21:08.559 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:08.559 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-29T20:21:08.613 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --addv c '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]' --add b 172.21.15.167 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-04-29T20:21:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-04-29T20:21:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:08.700 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-29T20:21:08.701 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-29T20:21:08.716 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc... 2024-04-29T20:21:08.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-29T20:21:08.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-29T20:21:08.797 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-04-29T20:21:08.797 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-04-29T20:21:08.797 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-29T20:21:08.798 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-29T20:21:08.799 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.800 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167" 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 1 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: osd heartbeat grace = 80 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: osd op thread timeout = 150 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:08.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-04-29T20:21:08.802 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-04-29T20:21:08.803 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: bdev async discard = True 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: bdev enable discard = True 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore allocator = bitmap 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-04-29T20:21:08.804 INFO:teuthology.orchestra.run.smithi064.stdout: osd class default list = * 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd class load list = * 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd client watch timeout = 120 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd fast shutdown = False 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-29T20:21:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-04-29T20:21:08.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.807 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd crush smoke test = False 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-04-29T20:21:08.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: debug objecter = 20 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout: debug rados = 20 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-04-29T20:21:08.809 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-04-29T20:21:08.811 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2024-04-29T20:21:08.811 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-29T20:21:08.812 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-29T20:21:08.813 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = "172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167" 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon client directed command retry = 5 2024-04-29T20:21:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: mon election default strategy = 1 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: ms inject socket failures = 5000 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: ms type = async 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd heartbeat grace = 80 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd op thread timeout = 150 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2024-04-29T20:21:08.815 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug trim objects = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2024-04-29T20:21:08.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bdev async discard = True 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bdev enable discard = True 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore allocator = bitmap 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore zero block detection = True 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1 2024-04-29T20:21:08.817 INFO:teuthology.orchestra.run.smithi167.stdout: debug monc = 20 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 10 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd class default list = * 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd class load list = * 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd client watch timeout = 120 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify cached snaps = True 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify missing on start = True 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd fast shutdown = False 2024-04-29T20:21:08.818 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock override recovery settings = True 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: osd op queue = debug_random 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: osd op queue cut off = debug_random 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: osd shutdown pgref assert = True 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-04-29T20:21:08.819 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-04-29T20:21:08.820 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-29T20:21:08.821 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: mon min osdmap epochs = 50 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd crush smoke test = False 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune interval = 2 2024-04-29T20:21:08.822 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune min = 15 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune txsize = 2 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: mon scrub interval = 300 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on pool no app = False 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: paxos service trim min = 10 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: debug monc = 20 2024-04-29T20:21:08.823 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-29T20:21:08.824 INFO:teuthology.orchestra.run.smithi167.stdout: debug objecter = 20 2024-04-29T20:21:08.824 INFO:teuthology.orchestra.run.smithi167.stdout: debug rados = 20 2024-04-29T20:21:08.824 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2024-04-29T20:21:08.824 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.c] 2024-04-29T20:21:08.824 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.b] 2024-04-29T20:21:08.824 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-29T20:21:08.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-29T20:21:08.905 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-29T20:21:08.905 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:08.906 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-29T20:21:08.925 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:08.925 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-29T20:21:08.980 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:21:08.980 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:08.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-29T20:21:08.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-29T20:21:09.055 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:09.055 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:09.114 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:21:09.114 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:09.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-29T20:21:09.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-29T20:21:09.150 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:09.150 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:09.205 INFO:tasks.ceph:Setting up mon nodes... 2024-04-29T20:21:09.205 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-29T20:21:09.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-29T20:21:09.255 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-29T20:21:09.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-29T20:21:09.340 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-29T20:21:09.342 INFO:tasks.ceph:Setting up mds nodes... 2024-04-29T20:21:09.342 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-29T20:21:09.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-29T20:21:09.384 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-29T20:21:09.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-29T20:21:09.436 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-29T20:21:09.448 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-29T20:21:09.448 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-29T20:21:09.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-29T20:21:09.473 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:09.473 INFO:tasks.ceph:role: osd.0 2024-04-29T20:21:09.473 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:21:09.473 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-29T20:21:09.545 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:09.545 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:09.545 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:09.546 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:09.546 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:09.546 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:09.546 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-29T20:21:10.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-29T20:21:10.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-29T20:21:10.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-29T20:21:10.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-29T20:21:10.270 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:10.270 INFO:tasks.ceph:role: osd.1 2024-04-29T20:21:10.270 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:21:10.270 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-29T20:21:10.342 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:10.342 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:10.343 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:10.343 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:10.343 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:10.821 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-29T20:21:10.823 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-29T20:21:10.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-29T20:21:10.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-29T20:21:10.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-29T20:21:11.039 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:11.039 INFO:tasks.ceph:role: osd.2 2024-04-29T20:21:11.039 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:21:11.039 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-29T20:21:11.110 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:11.628 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-29T20:21:11.630 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-29T20:21:11.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-29T20:21:11.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-29T20:21:11.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-29T20:21:11.838 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:11.838 INFO:tasks.ceph:role: osd.3 2024-04-29T20:21:11.838 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:21:11.838 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-29T20:21:11.910 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:11.910 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:11.911 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:11.911 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:11.911 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:12.425 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-29T20:21:12.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-29T20:21:12.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-29T20:21:12.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-29T20:21:12.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:12.682+0000 7f4c6f863640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-29T20:21:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:12.682+0000 7f4c6f863640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-29T20:21:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:12.682+0000 7f4c6f863640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-29T20:21:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:12.682+0000 7f4c6f863640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-29T20:21:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:12.682+0000 7f4c6f863640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-29T20:21:12.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:12.683+0000 7f4c6f863640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-29T20:21:15.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-29T20:21:15.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:15.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:15.650+0000 7f1bfac21640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-29T20:21:15.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:15.650+0000 7f1bfac21640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-29T20:21:15.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:15.650+0000 7f1bfac21640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-29T20:21:15.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:15.650+0000 7f1bfac21640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-29T20:21:15.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:15.650+0000 7f1bfac21640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-29T20:21:15.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:15.651+0000 7f1bfac21640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-29T20:21:18.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-29T20:21:18.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:18.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:18.627+0000 7f0b52815640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-29T20:21:18.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:18.627+0000 7f0b52815640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-29T20:21:18.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:18.627+0000 7f0b52815640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-29T20:21:18.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:18.627+0000 7f0b52815640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-29T20:21:18.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:18.627+0000 7f0b52815640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-29T20:21:18.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:18.628+0000 7f0b52815640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-29T20:21:21.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-29T20:21:21.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:21.598+0000 7fc88e187640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-29T20:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:21.598+0000 7fc88e187640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-29T20:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:21.598+0000 7fc88e187640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-29T20:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:21.598+0000 7fc88e187640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-29T20:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:21.598+0000 7fc88e187640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-29T20:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:21.599+0000 7fc88e187640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-29T20:21:24.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-29T20:21:24.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-29T20:21:24.513 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:24.513 INFO:tasks.ceph:role: osd.4 2024-04-29T20:21:24.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:21:24.513 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:24.587 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:25.106 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-29T20:21:25.108 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-29T20:21:25.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-29T20:21:25.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-29T20:21:25.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-29T20:21:25.358 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:25.358 INFO:tasks.ceph:role: osd.5 2024-04-29T20:21:25.358 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:21:25.358 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-29T20:21:25.430 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:25.430 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:25.431 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:25.885 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-29T20:21:25.888 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-29T20:21:25.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-29T20:21:25.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-29T20:21:26.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-29T20:21:26.081 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:26.081 INFO:tasks.ceph:role: osd.6 2024-04-29T20:21:26.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:21:26.081 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:26.155 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:26.156 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:26.156 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:26.651 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-29T20:21:26.653 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-29T20:21:26.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-29T20:21:26.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-29T20:21:26.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-29T20:21:26.897 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-29T20:21:26.897 INFO:tasks.ceph:role: osd.7 2024-04-29T20:21:26.897 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:21:26.898 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-29T20:21:26.971 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-29T20:21:26.971 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-29T20:21:26.972 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-29T20:21:27.464 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-29T20:21:27.467 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-29T20:21:27.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-29T20:21:27.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-29T20:21:27.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:27.722 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:27.721+0000 7fbded3f3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-29T20:21:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:27.722+0000 7fbded3f3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-29T20:21:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:27.722+0000 7fbded3f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-29T20:21:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:27.722+0000 7fbded3f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-29T20:21:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:27.722+0000 7fbded3f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-29T20:21:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:27.722+0000 7fbded3f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-29T20:21:30.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-29T20:21:30.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:30.680+0000 7f29cc223640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-29T20:21:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:30.680+0000 7f29cc223640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-29T20:21:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:30.681+0000 7f29cc223640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-29T20:21:30.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:30.681+0000 7f29cc223640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-29T20:21:30.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:30.681+0000 7f29cc223640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-29T20:21:30.683 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:30.681+0000 7f29cc223640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-29T20:21:33.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-29T20:21:33.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:33.641+0000 7fe020763640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-29T20:21:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:33.641+0000 7fe020763640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-29T20:21:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:33.641+0000 7fe020763640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-29T20:21:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:33.641+0000 7fe020763640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-29T20:21:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:33.641+0000 7fe020763640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-29T20:21:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:33.642+0000 7fe020763640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-29T20:21:36.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-29T20:21:36.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:36.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:36.602+0000 7f2169d01640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-29T20:21:36.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:36.602+0000 7f2169d01640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-29T20:21:36.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:36.602+0000 7f2169d01640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-29T20:21:36.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:36.602+0000 7f2169d01640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-29T20:21:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:36.602+0000 7f2169d01640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-29T20:21:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:36.603+0000 7f2169d01640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-29T20:21:39.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-29T20:21:39.482 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-29T20:21:39.482 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:39.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-29T20:21:39.512 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:39.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-29T20:21:39.581 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:39.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-29T20:21:39.650 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:39.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-29T20:21:39.717 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:39.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-29T20:21:39.783 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:39.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-29T20:21:39.812 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:39.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-29T20:21:39.879 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:39.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-29T20:21:39.944 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:39.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-29T20:21:40.011 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:40.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-29T20:21:40.076 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:40.076 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-29T20:21:40.093 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:40.093 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-29T20:21:40.132 INFO:tasks.ceph:Adding keys to all mons... 2024-04-29T20:21:40.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-29T20:21:40.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-29T20:21:40.175 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-04-29T20:21:40.175 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA1ATBmMuw2DxAA8wrId8m4HT7u9Y2uybA0eg== 2024-04-29T20:21:40.175 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-04-29T20:21:40.175 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA4ATBmLEi6KBAASupG1bdrdxvSySk0qnx4QA== 2024-04-29T20:21:40.175 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA7ATBmBpTTJhAA0uSCKXB7HGLa2J+Efpij8w== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA+ATBmVaxuJRAARTy2EqWWzrSHzUIdlqEvxw== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBBATBmQEy3IxAAr89//g5KVcBAoBYeI/u0Jw== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA1ATBmlpA3FBAAlU8O9faonrLYtFz67WcH5A== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBHATBm2NsJKxAAG7LoyIR6si6ybl2MiIvLAg== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBKATBmrRKdKBAAE9/Wd48Wz3Yv4PZ74nsJAA== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNATBmr79IJhAAtLAE/YxKL4hidPSzPT15dg== 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-04-29T20:21:40.176 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBQATBmp/X1IxAAiWuaiB5lIR+AYe8205G76g== 2024-04-29T20:21:40.177 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-04-29T20:21:40.177 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA1ATBm9uvgFhAAAOEcTn5bDbdT6zs+m2JBUA== 2024-04-29T20:21:40.177 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-04-29T20:21:40.177 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA1ATBmap8DGhAAQUshFmdIaKxpwvYJVB7T2A== 2024-04-29T20:21:40.194 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.y] 2024-04-29T20:21:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA1ATBmMuw2DxAA8wrId8m4HT7u9Y2uybA0eg== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA4ATBmLEi6KBAASupG1bdrdxvSySk0qnx4QA== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA7ATBmBpTTJhAA0uSCKXB7HGLa2J+Efpij8w== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA+ATBmVaxuJRAARTy2EqWWzrSHzUIdlqEvxw== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.3] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBBATBmQEy3IxAAr89//g5KVcBAoBYeI/u0Jw== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.x] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA1ATBmlpA3FBAAlU8O9faonrLYtFz67WcH5A== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.4] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBHATBm2NsJKxAAG7LoyIR6si6ybl2MiIvLAg== 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.5] 2024-04-29T20:21:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBKATBmrRKdKBAAE9/Wd48Wz3Yv4PZ74nsJAA== 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.6] 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBNATBmr79IJhAAtLAE/YxKL4hidPSzPT15dg== 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.7] 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBQATBmp/X1IxAAiWuaiB5lIR+AYe8205G76g== 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA1ATBm9uvgFhAAAOEcTn5bDbdT6zs+m2JBUA== 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout:[client.1] 2024-04-29T20:21:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA1ATBmap8DGhAAQUshFmdIaKxpwvYJVB7T2A== 2024-04-29T20:21:40.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-29T20:21:40.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-29T20:21:40.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-29T20:21:40.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-29T20:21:40.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-29T20:21:40.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-29T20:21:40.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-29T20:21:40.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-29T20:21:40.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-29T20:21:40.856 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-29T20:21:40.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-29T20:21:40.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-29T20:21:40.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-29T20:21:41.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-29T20:21:41.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-29T20:21:41.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-29T20:21:41.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-29T20:21:41.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-29T20:21:41.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-29T20:21:41.328 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:41.330 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-29T20:21:41.383 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-29T20:21:41.383 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-04-29T20:21:41.383 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-29T20:21:41.384 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-04-29T20:21:41.386 INFO:tasks.ceph.mon.a:Started 2024-04-29T20:21:41.386 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-04-29T20:21:41.386 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-29T20:21:41.386 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-04-29T20:21:41.388 INFO:tasks.ceph.mon.c:Started 2024-04-29T20:21:41.388 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-04-29T20:21:41.388 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-29T20:21:41.388 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-04-29T20:21:41.425 INFO:tasks.ceph.mon.b:Started 2024-04-29T20:21:41.426 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-29T20:21:41.426 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-29T20:21:41.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-04-29T20:21:41.429 INFO:tasks.ceph.mgr.y:Started 2024-04-29T20:21:41.429 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-29T20:21:41.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-04-29T20:21:41.431 INFO:tasks.ceph.mgr.x:Started 2024-04-29T20:21:41.431 INFO:tasks.ceph:Setting crush tunables to default 2024-04-29T20:21:41.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-04-29T20:21:41.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.532+0000 7f5250ef2640 10 monclient: get_monmap_and_config 2024-04-29T20:21:41.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.532+0000 7f5250ef2640 10 monclient: build_initial_monmap 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.532+0000 7f5250ef2640 10 monclient: monmap: 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:41.533426+0000 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:41.533426+0000 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:41.535 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:41.536 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:41.536 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:41.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.533+0000 7f5250ef2640 1 Processor -- start 2024-04-29T20:21:41.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.533+0000 7f5250ef2640 1 -- start start 2024-04-29T20:21:41.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.533+0000 7f5250ef2640 10 monclient: init 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.533+0000 7f5250ef2640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.533+0000 7f5250ef2640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.533+0000 7f5250ef2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f524c0b2230 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 0x7f524c0b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 10 monclient(hunting): picked mon.noname-b con 0x7f524c0aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f524c0b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:41.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524c0ac1f0 con 0x7f524c0b5c60 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524c0ac4d0 con 0x7f524c0b2230 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f524c0b3440 con 0x7f524c0aa460 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5250ef2640 10 monclient(hunting): authenticate will time out at 1450.693672s 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f524ad9d640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f524c0b5c60 legacy=0x7f524c14dce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 msgr2=0x7f524c0b2f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f524a59c640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f524c0b2230 legacy=0x7f524c146970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:41.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.534+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 0x7f524c0b2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:41.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.735+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 msgr2=0x7f524c0b2f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:41.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.735+0000 7f524ad9d640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f524c0b5c60 legacy=0x7f524c14dce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-04-29T20:21:41.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.735+0000 7f524a59c640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f524c0b2230 legacy=0x7f524c146970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:41.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:41.735+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 0x7f524c0b2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.136+0000 7f524ad9d640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f524c0b5c60 legacy=0x7f524c14dce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-04-29T20:21:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.136+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 msgr2=0x7f524c0b2f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:42.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.136+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 0x7f524c0b2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:42.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.136+0000 7f524a59c640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f524c0b2230 legacy=0x7f524c146970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:42.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.937+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 msgr2=0x7f524c0b2f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:42.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.937+0000 7f524ad9d640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f524c0b5c60 legacy=0x7f524c14dce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-04-29T20:21:42.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.937+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 0x7f524c0b2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-29T20:21:42.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:42.937+0000 7f524a59c640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f524c0b2230 legacy=0x7f524c146970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): tick 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): _check_auth_tickets 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 monclient(hunting): continuing hunt 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 msgr2=0x7f524c0b2f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f524c0aa460 0x7f524c0b2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f524c0b2230 legacy=0x7f524c146970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f524c0b5c60 legacy=0x7f524c14dce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient: _add_conns ranks=[4,3] 2024-04-29T20:21:44.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 0x7f522c005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-c con 0x7f522c0036d0 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 0x7f522c008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-a con 0x7f522c0061a0 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f524c0b3440 con 0x7f522c0061a0 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f522c008d20 con 0x7f522c0036d0 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5248d99640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5243fff640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 msgr2=0x7f522c008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f5243fff640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 0x7f522c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f524a59c640 1 -- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 msgr2=0x7f522c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.167:3300/0 2024-04-29T20:21:44.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.534+0000 7f524a59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 0x7f522c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:44.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.735+0000 7f5243fff640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 msgr2=0x7f522c008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:44.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.735+0000 7f5243fff640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 0x7f522c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:44.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.735+0000 7f524a59c640 1 -- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 msgr2=0x7f522c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.167:3300/0 2024-04-29T20:21:44.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:44.735+0000 7f524a59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 0x7f522c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:45.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.136+0000 7f5243fff640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 msgr2=0x7f522c008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:45.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.136+0000 7f524a59c640 1 -- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 msgr2=0x7f522c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.167:3300/0 2024-04-29T20:21:45.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.136+0000 7f5243fff640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 0x7f522c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:45.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.136+0000 7f524a59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 0x7f522c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:45.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.937+0000 7f5243fff640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 msgr2=0x7f522c008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:45.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.937+0000 7f524a59c640 1 -- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 msgr2=0x7f522c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.167:3300/0 2024-04-29T20:21:45.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.937+0000 7f5243fff640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 0x7f522c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-29T20:21:45.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:45.937+0000 7f524a59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 0x7f522c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-29T20:21:47.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): tick 2024-04-29T20:21:47.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): _check_auth_tickets 2024-04-29T20:21:47.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 monclient(hunting): continuing hunt 2024-04-29T20:21:47.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-29T20:21:47.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 -- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 msgr2=0x7f522c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c0036d0 0x7f522c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 msgr2=0x7f522c008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c0061a0 0x7f522c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f524ad9d640 1 -- reap_dead start 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-a con 0x7f522c003b40 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 0x7f522c006f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-b con 0x7f522c006b90 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f522c0074b0 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524c0aa8a0 con 0x7f522c0074b0 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f524c0b3440 con 0x7f522c003b40 2024-04-29T20:21:47.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f522c008d20 con 0x7f522c006b90 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5248d99640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f524a59c640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 msgr2=0x7f522c006f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 0x7f522c006f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f524a59c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:47.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.534+0000 7f524ad9d640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f522c0074b0 legacy=0x7f522c007910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:47.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.735+0000 7f524a59c640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:47.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.735+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 msgr2=0x7f522c006f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:47.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.735+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 0x7f522c006f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:47.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.735+0000 7f524a59c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:47.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:47.735+0000 7f524ad9d640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f522c0074b0 legacy=0x7f522c007910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:48.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.136+0000 7f524a59c640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:48.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.136+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 msgr2=0x7f522c006f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:48.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.136+0000 7f524a59c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:48.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.136+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 0x7f522c006f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:48.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.136+0000 7f524ad9d640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f522c0074b0 legacy=0x7f522c007910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:48.411 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:00:06.888 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:21:48.723 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:00:07.200 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:21:48.888 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:00:07.394 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:21:48.888 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:00:07.394 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:21:48.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.937+0000 7f5243fff640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 msgr2=0x7f522c006f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3301/0 2024-04-29T20:21:48.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.937+0000 7f524a59c640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.64:3300/0 2024-04-29T20:21:48.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.937+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 0x7f522c006f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-29T20:21:48.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.937+0000 7f524a59c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-29T20:21:48.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:48.937+0000 7f524ad9d640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f522c0074b0 legacy=0x7f522c007910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-04-29T20:21:49.219 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:00:07.725 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:21:49.219 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:00:07.725 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): tick 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): _check_auth_tickets 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 monclient(hunting): continuing hunt 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 -- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 msgr2=0x7f522c006f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c006b90 0x7f522c006f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 -- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:50.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7f522c0074b0 legacy=0x7f522c007910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient: _add_conns ranks=[4,0] 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-c con 0x7f522c008420 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f522c010570 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524c0ac1f0 con 0x7f522c010570 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f524c0b3440 con 0x7f522c008420 2024-04-29T20:21:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.534+0000 7f5248d99640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:50.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:50.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:35736/0 (socket says 172.21.15.64:35736) 2024-04-29T20:21:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 learned_addr learned my addr 172.21.15.64:0/2550705505 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:21:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 10 monclient(hunting): get_auth_request con 0x7f522c008420 auth_method 0 2024-04-29T20:21:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.630+0000 7f524a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:50.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.631+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-29T20:21:50.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.631+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:50.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.631+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:50.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.631+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:50.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.832+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:50.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 10 monclient(hunting): get_auth_request con 0x7f522c008420 auth_method 0 2024-04-29T20:21:50.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:50.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:50.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:50.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-29T20:21:50.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:50.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:50.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:50.833+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:51.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.020+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v1:172.21.15.64:6789/0 conn(0x7f522c010570 legacy=0x7f522c012960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-29T20:21:51.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.020+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v1:172.21.15.64:6789/0 conn(0x7f522c010570 legacy=0x7f522c012960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-29T20:21:51.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.020+0000 7f5243fff640 1 --1- 172.21.15.64:0/2550705505 >> v1:172.21.15.64:6789/0 conn(0x7f522c010570 0x7f522c012960 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-04-29T20:21:51.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.020+0000 7f5243fff640 1 --1- 172.21.15.64:0/2550705505 >> v1:172.21.15.64:6789/0 conn(0x7f522c010570 0x7f522c012960 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-04-29T20:21:51.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.020+0000 7f5249d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6789/0 2024-04-29T20:21:51.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.305+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:51.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.316+0000 7f524a59c640 10 monclient(hunting): get_auth_request con 0x7f522c008420 auth_method 0 2024-04-29T20:21:51.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.316+0000 7f524a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:51.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.316+0000 7f524a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:51.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.316+0000 7f524a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:51.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.317+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-29T20:21:51.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.317+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:51.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.317+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:51.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:51.317+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:52.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.119+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:52.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.119+0000 7f524a59c640 10 monclient(hunting): get_auth_request con 0x7f522c008420 auth_method 0 2024-04-29T20:21:52.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.119+0000 7f524a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:52.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.119+0000 7f524a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:52.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.119+0000 7f524a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:52.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.120+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-29T20:21:52.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.120+0000 7f524a59c640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:52.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.120+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:52.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:52.120+0000 7f524a59c640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-29T20:21:53.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): tick 2024-04-29T20:21:53.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): _check_auth_tickets 2024-04-29T20:21:53.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 monclient(hunting): continuing hunt 2024-04-29T20:21:53.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-29T20:21:53.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 msgr2=0x7f522c010030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:53.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c008420 0x7f522c010030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 -- 172.21.15.64:0/2550705505 >> v1:172.21.15.64:6789/0 conn(0x7f522c010570 legacy=0x7f522c012960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f522c010570 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 reap_dead start 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-a con 0x7f522c003b40 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): picked mon.noname-c con 0x7f522c0078c0 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 -- 172.21.15.64:0/2550705505 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f524c0aa8a0 con 0x7f522c010570 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 -- 172.21.15.64:0/2550705505 --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f524c0b3440 con 0x7f522c003b40 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:53.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 1 -- 172.21.15.64:0/2550705505 --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f522c008d20 con 0x7f522c0078c0 2024-04-29T20:21:53.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.534+0000 7f5248d99640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:53.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.535+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:53.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 10 monclient(hunting): get_auth_request con 0x7f522c0078c0 auth_method 0 2024-04-29T20:21:53.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:53.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:53.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:53.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-29T20:21:53.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:53.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:53.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.536+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:53.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.578+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:53.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.579+0000 7f5243fff640 10 monclient(hunting): get_auth_request con 0x7f522c003b40 auth_method 0 2024-04-29T20:21:53.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.579+0000 7f5243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:53.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.579+0000 7f5243fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:53.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.579+0000 7f5243fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:53.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.592+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-29T20:21:53.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.592+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:53.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.592+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:53.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.592+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:53.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.737+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:53.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.737+0000 7f524ad9d640 10 monclient(hunting): get_auth_request con 0x7f522c0078c0 auth_method 0 2024-04-29T20:21:53.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.737+0000 7f524ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:53.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.737+0000 7f524ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:53.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.737+0000 7f524ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:53.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.738+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-29T20:21:53.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.738+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:53.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.738+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:53.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.738+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.864+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.865+0000 7f5243fff640 10 monclient(hunting): get_auth_request con 0x7f522c003b40 auth_method 0 2024-04-29T20:21:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.865+0000 7f5243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.865+0000 7f5243fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.865+0000 7f5243fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.888+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-29T20:21:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.888+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 msgr2=0x7f522c006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.888+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:53.888+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-29T20:21:54.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.149+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:54.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.150+0000 7f524ad9d640 10 monclient(hunting): get_auth_request con 0x7f522c0078c0 auth_method 0 2024-04-29T20:21:54.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.150+0000 7f524ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:54.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.150+0000 7f524ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:54.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.150+0000 7f524ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:54.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.152+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-29T20:21:54.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.152+0000 7f524ad9d640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:54.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.152+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:54.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.152+0000 7f524ad9d640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-29T20:21:54.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.330+0000 7f5249d9b640 1 -- 172.21.15.64:0/2550705505 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837913432 0 0) 0x7f524c0aa8a0 con 0x7f522c010570 2024-04-29T20:21:54.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.330+0000 7f5249d9b640 10 monclient(hunting): _negotiate 2024-04-29T20:21:54.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.330+0000 7f5249d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:54.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.330+0000 7f5249d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:54.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.330+0000 7f5249d9b640 10 monclient(hunting): my global_id is 4101 2024-04-29T20:21:54.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.330+0000 7f5249d9b640 1 -- 172.21.15.64:0/2550705505 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5238003520 con 0x7f522c010570 2024-04-29T20:21:54.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.339+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.64:3300/0 conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:54.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.341+0000 7f5243fff640 10 monclient(hunting): get_auth_request con 0x7f522c003b40 auth_method 0 2024-04-29T20:21:54.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.341+0000 7f5243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:54.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.341+0000 7f5243fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:54.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.341+0000 7f5243fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:21:54.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.399+0000 7f5243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:54.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.399+0000 7f5243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:54.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.399+0000 7f5243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 msgr2=0x7f522c006b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 >> v1:172.21.15.64:6789/0 conn(0x7f522c010570 legacy=0x7f522c012960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 1 monclient: found mon.noname-a 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:21:54.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.405+0000 7f5243fff640 1 -- 172.21.15.64:0/2550705505 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c0ac4d0 con 0x7f522c003b40 2024-04-29T20:21:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.428+0000 7f5243fff640 1 --2- 172.21.15.64:0/2550705505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 0x7f522c006650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f522c0080e0 tx=0x7f523402c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b310861f378fd0f server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:54.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.474+0000 7f5249d9b640 1 -- 172.21.15.64:0/2550705505 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234039070 con 0x7f522c003b40 2024-04-29T20:21:54.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.474+0000 7f5249d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: dump: 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:54.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: _finish_auth 0 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: _check_auth_tickets 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 1 -- 172.21.15.64:0/2550705505 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5234034440 con 0x7f522c003b40 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 1 -- 172.21.15.64:0/2550705505 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52340347f0 con 0x7f522c003b40 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 5 monclient: authenticate success, global_id 4107 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 4 set_mon_vals no callback set 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5249d9b640 10 monclient: dump: 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:54.478 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:54.479 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:54.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 10 monclient: shutdown 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 -- 172.21.15.64:0/2550705505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 msgr2=0x7f522c006650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 --2- 172.21.15.64:0/2550705505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 0x7f522c006650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f522c0080e0 tx=0x7f523402c1f0 comp rx=0 tx=0).stop 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 -- 172.21.15.64:0/2550705505 shutdown_connections 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 --2- 172.21.15.64:0/2550705505 >> v2:172.21.15.167:3300/0 conn(0x7f522c0078c0 0x7f522c006b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 --2- 172.21.15.64:0/2550705505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 0x7f522c006650 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 -- 172.21.15.64:0/2550705505 >> 172.21.15.64:0/2550705505 conn(0x7f524c06c960 msgr2=0x7f524c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 -- 172.21.15.64:0/2550705505 shutdown_connections 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.475+0000 7f5250ef2640 1 -- 172.21.15.64:0/2550705505 wait complete. 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.476+0000 7f5250ef2640 10 monclient: build_initial_monmap 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.476+0000 7f5250ef2640 10 monclient: monmap: 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:21:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 librados: starting msgr at 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 librados: starting objecter 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 Processor -- start 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 -- start start 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 librados: setting wanted keys 2024-04-29T20:21:54.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 librados: calling monclient init 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient: init 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 0x7f524c1433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): picked mon.noname-a con 0x7f522c003b40 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 0x7f524c0b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): picked mon.noname-c con 0x7f522c0078c0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f522c010570 0x7f524c0b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): picked mon.noname-b con 0x7f522c010570 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f524c0b2790 con 0x7f522c003b40 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f524c0b2610 con 0x7f522c010570 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:54.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f524c0b2f00 con 0x7f522c0078c0 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.477+0000 7f5250ef2640 10 monclient(hunting): authenticate will time out at 1463.637315s 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 0x7f524c0b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 0x7f524c0b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:39536/0 (socket says 172.21.15.64:39536) 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 1 -- 172.21.15.64:0/803409528 learned_addr learned my addr 172.21.15.64:0/803409528 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 10 monclient(hunting): get_auth_request con 0x7f522c0078c0 auth_method 0 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.478+0000 7f5243fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.479+0000 7f5243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.479+0000 7f5243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:54.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.479+0000 7f5243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-04-29T20:21:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 1 -- 172.21.15.64:0/803409528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f522c010570 msgr2=0x7f524c0b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f522c010570 0x7f524c0b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 1 -- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 msgr2=0x7f524c1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 0x7f524c1433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:54.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:54.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 1 monclient: found mon.noname-c 2024-04-29T20:21:54.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:54.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:21:54.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.484+0000 7f5243fff640 1 -- 172.21.15.64:0/803409528 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c0b0150 con 0x7f522c0078c0 2024-04-29T20:21:54.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.487+0000 7f5243fff640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 0x7f524c0b0850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5234000ea0 tx=0x7f5234000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:54.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.487+0000 7f5242ffd640 10 client.?.objecter ms_handle_connect 0x7f522c0078c0 2024-04-29T20:21:54.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.487+0000 7f5242ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:54.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234042020 con 0x7f522c0078c0 2024-04-29T20:21:54.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:54.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: dump: 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: _finish_auth 0 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f523403a040 con 0x7f522c0078c0 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234039070 con 0x7f522c0078c0 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5242ffd640 10 monclient: dump: 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:54.539 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5250ef2640 5 monclient: authenticate success, global_id 4103 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f524959a640 4 set_mon_vals no callback set 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5250ef2640 10 monclient: _renew_subs 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5250ef2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524c0b1860 con 0x7f522c0078c0 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.535+0000 7f5250ef2640 10 client.4103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.536+0000 7f5250ef2640 10 monclient: _renew_subs 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.536+0000 7f5250ef2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.536+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f524c0ae240 con 0x7f522c0078c0 2024-04-29T20:21:54.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.536+0000 7f5250ef2640 1 librados: init done 2024-04-29T20:21:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.537+0000 7f5250ef2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.537+0000 7f5250ef2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.537+0000 7f5250ef2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:21:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.537+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f524c04fce0 con 0x7f522c0078c0 2024-04-29T20:21:54.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.552+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5234045020 con 0x7f522c0078c0 2024-04-29T20:21:54.619 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:54.618+0000 7f06d7015040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-29T20:21:54.675 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:54.673+0000 7f33fdc21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-29T20:21:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5234050730 con 0x7f522c0078c0 2024-04-29T20:21:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 10 client.4103.objecter ms_dispatch 0x7f524c000b90 osd_map(1..1 src has 1..1) 2024-04-29T20:21:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 3 client.4103.objecter handle_osd_map got epochs [1,1] > 0 2024-04-29T20:21:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 3 client.4103.objecter handle_osd_map decoding full epoch 1 2024-04-29T20:21:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 20 client.4103.objecter dump_active .. 0 homeless 2024-04-29T20:21:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f523404c030 con 0x7f522c0078c0 2024-04-29T20:21:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.745+0000 7f5242ffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:54.777 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:54.776+0000 7f06d7015040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-29T20:21:54.824 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:54.822+0000 7f33fdc21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-29T20:21:54.834 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:54.832+0000 7f06d7015040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-29T20:21:54.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.844+0000 7f5250ef2640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-29T20:21:54.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.844+0000 7f5250ef2640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-29T20:21:54.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.844+0000 7f5250ef2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:21:54.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:54.844+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f524c071f90 con 0x7f522c0078c0 2024-04-29T20:21:54.878 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:54.877+0000 7f33fdc21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-29T20:21:55.091 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:55.090+0000 7f06d7015040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-29T20:21:55.132 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:55.130+0000 7f33fdc21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-29T20:21:55.142 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:55.141+0000 7f06d7015040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-29T20:21:55.178 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:55.177+0000 7f33fdc21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-29T20:21:55.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.216+0000 7f5242ffd640 1 -- 172.21.15.64:0/803409528 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f5234005be0 con 0x7f522c0078c0 2024-04-29T20:21:55.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.216+0000 7f5242ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-29T20:21:55.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.216+0000 7f5242ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-29T20:21:55.218 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 10 librados: watch_flush enter 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 10 librados: watch_flush exit 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 20 client.4103.objecter shutdown clearing up homeless session... 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 10 client.4103.objecter successfully canceled tick 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 10 monclient: shutdown 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 msgr2=0x7f524c0b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 0x7f524c0b0850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5234000ea0 tx=0x7f5234000bc0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 shutdown_connections 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f522c0078c0 0x7f524c0b0850 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f522c010570 0x7f524c0b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 --2- 172.21.15.64:0/803409528 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f522c003b40 0x7f524c1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.217+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 >> 172.21.15.64:0/803409528 conn(0x7f524c06c960 msgr2=0x7f524c0b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:55.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.218+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 shutdown_connections 2024-04-29T20:21:55.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.218+0000 7f5250ef2640 1 -- 172.21.15.64:0/803409528 wait complete. 2024-04-29T20:21:55.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:55.218+0000 7f5250ef2640 1 librados: shutdown 2024-04-29T20:21:55.231 INFO:tasks.ceph:check_enable_crimson: False 2024-04-29T20:21:55.231 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-29T20:21:55.231 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:55.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-29T20:21:55.259 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:55.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-29T20:21:55.327 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:55.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-29T20:21:55.392 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:21:55.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-29T20:21:55.456 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:55.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-29T20:21:55.481 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:55.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-29T20:21:55.544 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:55.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-29T20:21:55.608 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-29T20:21:55.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-29T20:21:55.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 5ab5e452-c2c0-4817-81db-f5fc686af04c 0 2024-04-29T20:21:55.809 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.808+0000 7fd7b181b640 10 monclient: get_monmap_and_config 2024-04-29T20:21:55.809 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.808+0000 7fd7b181b640 10 monclient: build_initial_monmap 2024-04-29T20:21:55.809 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.808+0000 7fd7b181b640 10 monclient: monmap: 2024-04-29T20:21:55.809 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:55.809 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:55.809 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:55.808955+0000 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:55.808955+0000 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:55.810 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.808+0000 7fd7b181b640 1 Processor -- start 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.808+0000 7fd7b181b640 1 -- start start 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.808+0000 7fd7b181b640 10 monclient: init 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 0x7fd7ac147310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac06c350 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd7ac0af990 0x7fd7ac1409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0af990 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7ac143490 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ac146e60 con 0x7fd7ac143490 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac0b1000 con 0x7fd7ac06c350 2024-04-29T20:21:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac0aecd0 con 0x7fd7ac0af990 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.809+0000 7fd7b181b640 10 monclient(hunting): authenticate will time out at 1406.198825s 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aa7fc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd7ac0af990 0x7fd7ac1409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 0x7fd7ac147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7ab7fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fd7ac143490 0x7fd7ac143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57180/0 (socket says 172.21.15.167:57180) 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7ab7fe640 1 -- 172.21.15.167:0/2697973350 learned_addr learned my addr 172.21.15.167:0/2697973350 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 10 monclient(hunting): get_auth_request con 0x7fd7ac06c350 auth_method 0 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aa7fc640 10 monclient(hunting): get_auth_request con 0x7fd7ac0af990 auth_method 0 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:55.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 1 -- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 msgr2=0x7fd7ac147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 1 -- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 msgr2=0x7fd7ac147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 1 --2- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 0x7fd7ac147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.810+0000 7fd7aaffd640 1 --2- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 0x7fd7ac147310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.811+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.811+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.811+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 1 -- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 msgr2=0x7fd7ac147310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 1 --2- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 0x7fd7ac147310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 1 -- 172.21.15.167:0/2697973350 >> v1:172.21.15.167:6789/0 conn(0x7fd7ac143490 legacy=0x7fd7ac143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 10 monclient: _finish_hunting 0 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 1 monclient: found mon.noname-c 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:21:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.813+0000 7fd7aa7fc640 1 -- 172.21.15.167:0/2697973350 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac0aefb0 con 0x7fd7ac0af990 2024-04-29T20:21:55.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.816+0000 7fd7aa7fc640 1 --2- 172.21.15.167:0/2697973350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac0af990 0x7fd7ac1409f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac06dab0 tx=0x7fd7a002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=552a00729c60168d server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 1 -- 172.21.15.167:0/2697973350 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00315a0 con 0x7fd7ac0af990 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: dump: 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:55.823 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: _finish_auth 0 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 1 -- 172.21.15.167:0/2697973350 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a0037070 con 0x7fd7ac0af990 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 1 -- 172.21.15.167:0/2697973350 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0031960 con 0x7fd7ac0af990 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7b181b640 5 monclient: authenticate success, global_id 4111 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7b181b640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7b181b640 4 set_mon_vals no callback set 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.821+0000 7fd7a9ffb640 10 monclient: dump: 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:55.824 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:55.825 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:55.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 10 monclient: shutdown 2024-04-29T20:21:55.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 -- 172.21.15.167:0/2697973350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac0af990 msgr2=0x7fd7ac1409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 --2- 172.21.15.167:0/2697973350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac0af990 0x7fd7ac1409f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac06dab0 tx=0x7fd7a002b840 comp rx=0 tx=0).stop 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 -- 172.21.15.167:0/2697973350 shutdown_connections 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 --2- 172.21.15.167:0/2697973350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac0af990 0x7fd7ac1409f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 --2- 172.21.15.167:0/2697973350 >> v2:172.21.15.64:3300/0 conn(0x7fd7ac06c350 0x7fd7ac147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 -- 172.21.15.167:0/2697973350 >> 172.21.15.167:0/2697973350 conn(0x7fd7ac0b6bf0 msgr2=0x7fd7ac0b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 -- 172.21.15.167:0/2697973350 shutdown_connections 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.822+0000 7fd7b181b640 1 -- 172.21.15.167:0/2697973350 wait complete. 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 10 monclient: build_initial_monmap 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 10 monclient: monmap: 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:55.829 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 1 librados: starting msgr at 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 1 librados: starting objecter 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 1 Processor -- start 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 1 -- start start 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 1 librados: setting wanted keys 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 1 librados: calling monclient init 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.823+0000 7fd7b181b640 10 monclient: init 2024-04-29T20:21:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac06c350 0x7fd7ac145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac06c350 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd7ac0af990 0x7fd7ac13bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0af990 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 0x7fd7ac13a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac143490 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac147850 con 0x7fd7ac06c350 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac0af570 con 0x7fd7ac143490 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac140f30 con 0x7fd7ac0af990 2024-04-29T20:21:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7b181b640 10 monclient(hunting): authenticate will time out at 1406.213833s 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.824+0000 7fd7ab7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 0x7fd7ac13a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 0x7fd7ac13a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59482/0 (socket says 172.21.15.167:59482) 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 1 -- 172.21.15.167:0/286790401 learned_addr learned my addr 172.21.15.167:0/286790401 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): get_auth_request con 0x7fd7ac143490 auth_method 0 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aa7fc640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd7ac0af990 0x7fd7ac13bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aaffd640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac06c350 0x7fd7ac145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aaffd640 10 monclient(hunting): get_auth_request con 0x7fd7ac06c350 auth_method 0 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aaffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aa7fc640 10 monclient(hunting): get_auth_request con 0x7fd7ac0af990 auth_method 0 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.825+0000 7fd7ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.826+0000 7fd7aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.826+0000 7fd7aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.826+0000 7fd7aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.826+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.826+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.826+0000 7fd7aa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 1 -- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd7ac0af990 msgr2=0x7fd7ac13bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd7ac0af990 0x7fd7ac13bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 1 -- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac06c350 msgr2=0x7fd7ac145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac06c350 0x7fd7ac145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:21:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 1 monclient: found mon.noname-b 2024-04-29T20:21:55.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:55.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:55.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.828+0000 7fd7ab7fe640 1 -- 172.21.15.167:0/286790401 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac071100 con 0x7fd7ac143490 2024-04-29T20:21:55.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.831+0000 7fd7ab7fe640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 0x7fd7ac13a390 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd7a400c9f0 tx=0x7fd7a400ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:55.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.831+0000 7fd793fff640 10 client.?.objecter ms_handle_connect 0x7fd7ac143490 2024-04-29T20:21:55.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.831+0000 7fd793fff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4017070 con 0x7fd7ac143490 2024-04-29T20:21:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: dump: 2024-04-29T20:21:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: _finish_auth 0 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: _check_auth_tickets 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a400ab70 con 0x7fd7ac143490 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:55.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4012400 con 0x7fd7ac143490 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 5 monclient: authenticate success, global_id 4114 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 10 monclient: _renew_subs 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac071e50 con 0x7fd7ac143490 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7a97fa640 4 set_mon_vals no callback set 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd793fff640 10 monclient: dump: 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 10 monclient: _renew_subs 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac13a8d0 con 0x7fd7ac143490 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.835+0000 7fd7b181b640 1 librados: init done 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.836+0000 7fd7b181b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.836+0000 7fd7b181b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.836+0000 7fd7b181b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:55.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:55.836+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ac04fce0 con 0x7fd7ac143490 2024-04-29T20:21:55.877 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:55.876+0000 7f33fdc21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-29T20:21:55.905 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:55.903+0000 7f06d7015040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-29T20:21:55.968 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:55.966+0000 7f33fdc21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-29T20:21:55.995 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:55.994+0000 7f06d7015040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-29T20:21:56.011 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:56.010+0000 7f33fdc21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-29T20:21:56.014 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd7a401a070 con 0x7fd7ac143490 2024-04-29T20:21:56.014 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fd7a401c9e0 con 0x7fd7ac143490 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 10 client.4114.objecter ms_dispatch 0x7fd7ac000b90 osd_map(2..2 src has 1..2) 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 20 client.4114.objecter dump_active .. 0 homeless 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd7a40070d0 con 0x7fd7ac143490 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.013+0000 7fd793fff640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:56.040 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:56.038+0000 7f06d7015040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-29T20:21:56.087 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.085+0000 7fd7b181b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5ab5e452-c2c0-4817-81db-f5fc686af04c", "id": 0}] 2024-04-29T20:21:56.087 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.085+0000 7fd7b181b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5ab5e452-c2c0-4817-81db-f5fc686af04c", "id": 0}] 2024-04-29T20:21:56.087 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.085+0000 7fd7b181b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:56.087 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.085+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5ab5e452-c2c0-4817-81db-f5fc686af04c", "id": 0} v 0) -- 0x7fd7ac06dea0 con 0x7fd7ac143490 2024-04-29T20:21:56.183 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:56.181+0000 7f33fdc21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-29T20:21:56.214 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:56.213+0000 7f06d7015040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-29T20:21:56.231 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:56.229+0000 7f33fdc21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-29T20:21:56.278 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:56.276+0000 7f06d7015040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-29T20:21:56.285 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:56.284+0000 7f33fdc21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-29T20:21:56.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd793fff640 1 -- 172.21.15.167:0/286790401 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ab5e452-c2c0-4817-81db-f5fc686af04c", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fd7a40127e0 con 0x7fd7ac143490 2024-04-29T20:21:56.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd793fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5ab5e452-c2c0-4817-81db-f5fc686af04c", "id": 0}] 2024-04-29T20:21:56.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd793fff640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:56.309 INFO:teuthology.orchestra.run.smithi167.stdout:0 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd7b181b640 10 librados: watch_flush enter 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd7b181b640 10 librados: watch_flush exit 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd7b181b640 20 client.4114.objecter shutdown clearing up homeless session... 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.308+0000 7fd7b181b640 10 client.4114.objecter successfully canceled tick 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 10 monclient: shutdown 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 msgr2=0x7fd7ac13a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 0x7fd7ac13a390 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd7a400c9f0 tx=0x7fd7a400ceb0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 shutdown_connections 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd7ac0af990 0x7fd7ac13bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7ac143490 0x7fd7ac13a390 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 --2- 172.21.15.167:0/286790401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7ac06c350 0x7fd7ac145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 >> 172.21.15.167:0/286790401 conn(0x7fd7ac0b6bf0 msgr2=0x7fd7ac0b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 shutdown_connections 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 -- 172.21.15.167:0/286790401 wait complete. 2024-04-29T20:21:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.309+0000 7fd7b181b640 1 librados: shutdown 2024-04-29T20:21:56.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new c736ff85-13c2-4c71-b5e3-b603c9023744 1 2024-04-29T20:21:56.337 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:56.336+0000 7f06d7015040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-29T20:21:56.406 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:56.405+0000 7f33fdc21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.419+0000 7f6bc46c3640 10 monclient: get_monmap_and_config 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.419+0000 7f6bc46c3640 10 monclient: build_initial_monmap 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.419+0000 7f6bc46c3640 10 monclient: monmap: 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:56.420638+0000 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:56.420638+0000 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:56.422 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.420+0000 7f6bc46c3640 1 Processor -- start 2024-04-29T20:21:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.420+0000 7f6bc46c3640 1 -- start start 2024-04-29T20:21:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.420+0000 7f6bc46c3640 10 monclient: init 2024-04-29T20:21:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-29T20:21:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient(hunting): picked mon.noname-b con 0x7f6bbc0b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bbc0aa460 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bbc0b5c60 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bbc146eb0 con 0x7f6bbc0aa460 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bbc0ac1f0 con 0x7f6bbc0b5c60 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f6bbc0ac4d0 con 0x7f6bbc0b2230 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc46c3640 10 monclient(hunting): authenticate will time out at 1406.810707s 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc2c60640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f6bbc0b5c60 0x7f6bbc14dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57184/0 (socket says 172.21.15.167:57184) 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.421+0000 7f6bc2c60640 1 -- 172.21.15.167:0/1784481392 learned_addr learned my addr 172.21.15.167:0/1784481392 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.422+0000 7f6bc245f640 1 --2- 172.21.15.167:0/1784481392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.422+0000 7f6bc245f640 10 monclient(hunting): get_auth_request con 0x7f6bbc0b2230 auth_method 0 2024-04-29T20:21:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.422+0000 7f6bc245f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.422+0000 7f6bc245f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.422+0000 7f6bc245f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.423+0000 7f6bc245f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.423+0000 7f6bc245f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.423+0000 7f6bc245f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410743095 0 0) 0x7f6bbc146eb0 con 0x7f6bbc0aa460 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): _negotiate 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): my global_id is 4119 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ba8003520 con 0x7f6bbc0aa460 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863299189 0 0) 0x7f6bbc0ac1f0 con 0x7f6bbc0b5c60 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): _negotiate 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 10 monclient(hunting): my global_id is 4117 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.424+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bbc146eb0 con 0x7f6bbc0b5c60 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 1 -- 172.21.15.167:0/1784481392 >> v1:172.21.15.167:6789/0 conn(0x7f6bbc0b5c60 legacy=0x7f6bbc14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 1 -- 172.21.15.167:0/1784481392 >> v1:172.21.15.64:6789/0 conn(0x7f6bbc0aa460 legacy=0x7f6bbc0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 10 monclient: _finish_hunting 0 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 1 monclient: found mon.noname-b 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:21:56.428 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.425+0000 7f6bc245f640 1 -- 172.21.15.167:0/1784481392 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bbc0aa8a0 con 0x7f6bbc0b2230 2024-04-29T20:21:56.429 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.428+0000 7f6bc245f640 1 --2- 172.21.15.167:0/1784481392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc146970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6bac0097d0 tx=0x7f6bac02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f647447fab5eba server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac035020 con 0x7f6bbc0b2230 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: dump: 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: _finish_auth 0 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: _check_auth_tickets 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc46c3640 5 monclient: authenticate success, global_id 4109 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bac031890 con 0x7f6bbc0b2230 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc46c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc46c3640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc46c3640 4 set_mon_vals no callback set 2024-04-29T20:21:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 1 -- 172.21.15.167:0/1784481392 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac031c60 con 0x7f6bbc0b2230 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.433+0000 7f6bc145d640 10 monclient: dump: 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 10 monclient: shutdown 2024-04-29T20:21:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1784481392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 msgr2=0x7f6bbc146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 --2- 172.21.15.167:0/1784481392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc146970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6bac0097d0 tx=0x7f6bac02b930 comp rx=0 tx=0).stop 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1784481392 shutdown_connections 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 --2- 172.21.15.167:0/1784481392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc146970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1784481392 >> 172.21.15.167:0/1784481392 conn(0x7f6bbc06c960 msgr2=0x7f6bbc06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1784481392 shutdown_connections 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1784481392 wait complete. 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 10 monclient: build_initial_monmap 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.434+0000 7f6bc46c3640 10 monclient: monmap: 2024-04-29T20:21:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 librados: starting msgr at 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 librados: starting objecter 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 Processor -- start 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 -- start start 2024-04-29T20:21:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 librados: setting wanted keys 2024-04-29T20:21:56.441 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 librados: calling monclient init 2024-04-29T20:21:56.441 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient: init 2024-04-29T20:21:56.443 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 0x7f6bbc0ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): picked mon.noname-b con 0x7f6bbc0aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc0ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): picked mon.noname-c con 0x7f6bbc0b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bbc0b5c60 0x7f6bbc1dc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): picked mon.noname-a con 0x7f6bbc0b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6bbc0b2790 con 0x7f6bbc0b5c60 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6bbc0b2610 con 0x7f6bbc0aa460 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f6bbc14f400 con 0x7f6bbc0b2230 2024-04-29T20:21:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.435+0000 7f6bc46c3640 10 monclient(hunting): authenticate will time out at 1406.825218s 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 0x7f6bbc0ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc1c5e640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc0ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc2c60640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bbc0b5c60 0x7f6bbc1dc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 0x7f6bbc0ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59488/0 (socket says 172.21.15.167:59488) 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 1 -- 172.21.15.167:0/1378664626 learned_addr learned my addr 172.21.15.167:0/1378664626 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 10 monclient(hunting): get_auth_request con 0x7f6bbc0aa460 auth_method 0 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc245f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc2c60640 10 monclient(hunting): get_auth_request con 0x7f6bbc0b5c60 auth_method 0 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc2c60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc2c60640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc2c60640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc1c5e640 10 monclient(hunting): get_auth_request con 0x7f6bbc0b2230 auth_method 0 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc1c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc1c5e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.436+0000 7f6bc1c5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc245f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc245f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc245f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc2c60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc2c60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc2c60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc1c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc1c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.437+0000 7f6bc1c5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 1 -- 172.21.15.167:0/1378664626 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 msgr2=0x7f6bbc0ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc0ad1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 1 -- 172.21.15.167:0/1378664626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bbc0b5c60 msgr2=0x7f6bbc1dc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bbc0b5c60 0x7f6bbc1dc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 10 monclient: _finish_hunting 0 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 1 monclient: found mon.noname-b 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.440+0000 7f6bc245f640 1 -- 172.21.15.167:0/1378664626 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bbc0aea70 con 0x7f6bbc0aa460 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.442+0000 7f6bc245f640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 0x7f6bbc0ac940 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6bac02bf40 tx=0x7f6bac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.442+0000 7f6bb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6bbc0aa460 2024-04-29T20:21:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.442+0000 7f6bb2ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:56.448 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac035040 con 0x7f6bbc0aa460 2024-04-29T20:21:56.448 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.448 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: dump: 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: _finish_auth 0 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bc46c3640 5 monclient: authenticate success, global_id 4120 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bac030c90 con 0x7f6bbc0aa460 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac044670 con 0x7f6bbc0aa460 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bc46c3640 10 monclient: _renew_subs 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bc46c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bbc13df60 con 0x7f6bbc0aa460 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bc0c5c640 4 set_mon_vals no callback set 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.445+0000 7f6bb2ffd640 10 monclient: dump: 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.446+0000 7f6bc46c3640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.446+0000 7f6bc46c3640 10 monclient: _renew_subs 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.446+0000 7f6bc46c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.446+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bbc1dc580 con 0x7f6bbc0aa460 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.446+0000 7f6bc46c3640 1 librados: init done 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.447+0000 7f6bc46c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.447+0000 7f6bc46c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.447+0000 7f6bc46c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.447+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bbc04fce0 con 0x7f6bbc0aa460 2024-04-29T20:21:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.448+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6bac038020 con 0x7f6bbc0aa460 2024-04-29T20:21:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.448+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f6bac04d6b0 con 0x7f6bbc0aa460 2024-04-29T20:21:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.448+0000 7f6bb2ffd640 10 client.4120.objecter ms_dispatch 0x7f6bbc000b90 osd_map(3..3 src has 1..3) 2024-04-29T20:21:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.448+0000 7f6bb2ffd640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-04-29T20:21:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.448+0000 7f6bb2ffd640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-04-29T20:21:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.448+0000 7f6bb2ffd640 20 client.4120.objecter dump_active .. 0 homeless 2024-04-29T20:21:56.466 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:56.465+0000 7f06d7015040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-29T20:21:56.552 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.550+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6bac02d040 con 0x7f6bbc0aa460 2024-04-29T20:21:56.552 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.551+0000 7f6bb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.552 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.551+0000 7f6bb2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:56.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.623+0000 7f6bc46c3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c736ff85-13c2-4c71-b5e3-b603c9023744", "id": 1}] 2024-04-29T20:21:56.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.623+0000 7f6bc46c3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c736ff85-13c2-4c71-b5e3-b603c9023744", "id": 1}] 2024-04-29T20:21:56.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.623+0000 7f6bc46c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:56.625 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.623+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c736ff85-13c2-4c71-b5e3-b603c9023744", "id": 1} v 0) -- 0x7f6bbc1dbbf0 con 0x7f6bbc0aa460 2024-04-29T20:21:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.703+0000 7f6bb2ffd640 1 -- 172.21.15.167:0/1378664626 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c736ff85-13c2-4c71-b5e3-b603c9023744", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f6bac031750 con 0x7f6bbc0aa460 2024-04-29T20:21:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.703+0000 7f6bb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c736ff85-13c2-4c71-b5e3-b603c9023744", "id": 1}] 2024-04-29T20:21:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.703+0000 7f6bb2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:56.705 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-04-29T20:21:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 10 librados: watch_flush enter 2024-04-29T20:21:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 10 librados: watch_flush exit 2024-04-29T20:21:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 20 client.4120.objecter shutdown clearing up homeless session... 2024-04-29T20:21:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 10 client.4120.objecter successfully canceled tick 2024-04-29T20:21:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 10 monclient: shutdown 2024-04-29T20:21:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 msgr2=0x7f6bbc0ac940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 0x7f6bbc0ac940 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6bac02bf40 tx=0x7f6bac0046c0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 shutdown_connections 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f6bbc0b2230 0x7f6bbc0ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bbc0b5c60 0x7f6bbc1dc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 --2- 172.21.15.167:0/1378664626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bbc0aa460 0x7f6bbc0ac940 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 >> 172.21.15.167:0/1378664626 conn(0x7f6bbc06c960 msgr2=0x7f6bbc0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 shutdown_connections 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 -- 172.21.15.167:0/1378664626 wait complete. 2024-04-29T20:21:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.705+0000 7f6bc46c3640 1 librados: shutdown 2024-04-29T20:21:56.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 2d380b76-312c-43ca-8301-7ec705d9fca3 2 2024-04-29T20:21:56.805 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:56.804+0000 7f33fdc21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-29T20:21:56.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.813+0000 7f1751bc7640 10 monclient: get_monmap_and_config 2024-04-29T20:21:56.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.813+0000 7f1751bc7640 10 monclient: build_initial_monmap 2024-04-29T20:21:56.814 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.813+0000 7f1751bc7640 10 monclient: monmap: 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:56.814267+0000 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:56.814267+0000 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:56.815 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 1 Processor -- start 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 1 -- start start 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 10 monclient: init 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f174c0b2230 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f174c0aa460 0x7f174c0b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 10 monclient(hunting): picked mon.noname-a con 0x7f174c0aa460 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.814+0000 7f1751bc7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f174c0b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f1751bc7640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f174c0ac1f0 con 0x7f174c0b5c60 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f1751bc7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f174c0ac4d0 con 0x7f174c0b2230 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f1751bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f1751bc7640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f174c0b3440 con 0x7f174c0aa460 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f1751bc7640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f1751bc7640 10 monclient(hunting): authenticate will time out at 1407.204414s 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f174b7fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f174c0b2230 0x7f174c146970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57186/0 (socket says 172.21.15.167:57186) 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f174b7fe640 1 -- 172.21.15.167:0/493720104 learned_addr learned my addr 172.21.15.167:0/493720104 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:56.818 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.815+0000 7f174affd640 1 --2- 172.21.15.167:0/493720104 >> v2:172.21.15.64:3300/0 conn(0x7f174c0aa460 0x7f174c0b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.816+0000 7f174affd640 10 monclient(hunting): get_auth_request con 0x7f174c0aa460 auth_method 0 2024-04-29T20:21:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.816+0000 7f174affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.816+0000 7f174affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.816+0000 7f174affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.817+0000 7f174affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.817+0000 7f174affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.817+0000 7f174affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.818+0000 7f174a7fc640 1 -- 172.21.15.167:0/493720104 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4275713757 0 0) 0x7f174c0ac4d0 con 0x7f174c0b2230 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.818+0000 7f174a7fc640 10 monclient(hunting): _negotiate 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.818+0000 7f174a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.818+0000 7f174a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.818+0000 7f174a7fc640 10 monclient(hunting): my global_id is 4123 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.818+0000 7f174a7fc640 1 -- 172.21.15.167:0/493720104 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1734003520 con 0x7f174c0b2230 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 1 -- 172.21.15.167:0/493720104 >> v1:172.21.15.167:6789/0 conn(0x7f174c0b2230 legacy=0x7f174c146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 1 -- 172.21.15.167:0/493720104 >> v1:172.21.15.64:6789/0 conn(0x7f174c0b5c60 legacy=0x7f174c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 10 monclient: _finish_hunting 0 2024-04-29T20:21:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 1 monclient: found mon.noname-a 2024-04-29T20:21:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:21:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.819+0000 7f174affd640 1 -- 172.21.15.167:0/493720104 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c0aa8d0 con 0x7f174c0aa460 2024-04-29T20:21:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.823+0000 7f174affd640 1 --2- 172.21.15.167:0/493720104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0aa460 0x7f174c0b2f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17400097a0 tx=0x7f174002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a68190546e61d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:56.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 1 -- 172.21.15.167:0/493720104 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740035020 con 0x7f174c0aa460 2024-04-29T20:21:56.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: dump: 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: _finish_auth 0 2024-04-29T20:21:56.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: _check_auth_tickets 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 1 -- 172.21.15.167:0/493720104 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1740031830 con 0x7f174c0aa460 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f1751bc7640 5 monclient: authenticate success, global_id 4125 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f1751bc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f1751bc7640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 1 -- 172.21.15.167:0/493720104 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740031c00 con 0x7f174c0aa460 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f1751bc7640 4 set_mon_vals no callback set 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.827+0000 7f174a7fc640 10 monclient: dump: 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.830 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.831 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 10 monclient: shutdown 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 -- 172.21.15.167:0/493720104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0aa460 msgr2=0x7f174c0b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 --2- 172.21.15.167:0/493720104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0aa460 0x7f174c0b2f00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17400097a0 tx=0x7f174002b8d0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 -- 172.21.15.167:0/493720104 shutdown_connections 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 --2- 172.21.15.167:0/493720104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0aa460 0x7f174c0b2f00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 -- 172.21.15.167:0/493720104 >> 172.21.15.167:0/493720104 conn(0x7f174c06c960 msgr2=0x7f174c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 -- 172.21.15.167:0/493720104 shutdown_connections 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.828+0000 7f1751bc7640 1 -- 172.21.15.167:0/493720104 wait complete. 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 10 monclient: build_initial_monmap 2024-04-29T20:21:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 10 monclient: monmap: 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.836 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 1 librados: starting msgr at 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 1 librados: starting objecter 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 1 Processor -- start 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 1 -- start start 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 1 librados: setting wanted keys 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 1 librados: calling monclient init 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.829+0000 7f1751bc7640 10 monclient: init 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f174c0aa460 0x7f174c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): picked mon.noname-b con 0x7f174c0aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0b2230 0x7f174c070a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): picked mon.noname-a con 0x7f174c0b2230 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f174c0b5c60 0x7f174c1e05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): picked mon.noname-c con 0x7f174c0b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f174c0b6210 con 0x7f174c0b2230 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f174c0b6090 con 0x7f174c0aa460 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f174c14f400 con 0x7f174c0b5c60 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f1751bc7640 10 monclient(hunting): authenticate will time out at 1407.219721s 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f174c0aa460 0x7f174c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.838 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.830+0000 7f174affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0b2230 0x7f174c070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174bfff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f174c0b5c60 0x7f174c1e05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f174c0aa460 0x7f174c070550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59500/0 (socket says 172.21.15.167:59500) 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 1 -- 172.21.15.167:0/1553604309 learned_addr learned my addr 172.21.15.167:0/1553604309 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): get_auth_request con 0x7f174c0aa460 auth_method 0 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): get_auth_request con 0x7f174c0b2230 auth_method 0 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174bfff640 10 monclient(hunting): get_auth_request con 0x7f174c0b5c60 auth_method 0 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:56.839 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.832+0000 7f174bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.832+0000 7f174bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.832+0000 7f174bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 10 monclient(hunting): handle_auth_done global_id 4131 payload 824 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 1 -- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f174c0b5c60 msgr2=0x7f174c1e05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f174c0b5c60 0x7f174c1e05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 1 -- 172.21.15.167:0/1553604309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f174c0aa460 msgr2=0x7f174c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f174c0aa460 0x7f174c070550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 10 monclient: _finish_hunting 0 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 1 monclient: found mon.noname-a 2024-04-29T20:21:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:56.842 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:21:56.842 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.834+0000 7f174affd640 1 -- 172.21.15.167:0/1553604309 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c06df40 con 0x7f174c0b2230 2024-04-29T20:21:56.842 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.837+0000 7f174affd640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0b2230 0x7f174c070a90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1740033000 tx=0x7f17400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:56.842 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.837+0000 7f1733fff640 10 client.?.objecter ms_handle_connect 0x7f174c0b2230 2024-04-29T20:21:56.842 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.837+0000 7f1733fff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:56.843 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740040070 con 0x7f174c0b2230 2024-04-29T20:21:56.843 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.843 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: dump: 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: _finish_auth 0 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: _check_auth_tickets 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f174003a910 con 0x7f174c0b2230 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 5 monclient: authenticate success, global_id 4131 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 10 monclient: _renew_subs 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740035040 con 0x7f174c0b2230 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c06fe50 con 0x7f174c0b2230 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1749ffb640 4 set_mon_vals no callback set 2024-04-29T20:21:56.845 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1733fff640 10 monclient: dump: 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 10 monclient: _renew_subs 2024-04-29T20:21:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:56.847 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c13b8a0 con 0x7f174c0b2230 2024-04-29T20:21:56.847 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.841+0000 7f1751bc7640 1 librados: init done 2024-04-29T20:21:56.847 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.842+0000 7f1751bc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.847 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.842+0000 7f1751bc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.847 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.842+0000 7f1751bc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:56.847 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.842+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174c04fce0 con 0x7f174c0b2230 2024-04-29T20:21:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.858+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1740038020 con 0x7f174c0b2230 2024-04-29T20:21:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.858+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f1740039850 con 0x7f174c0b2230 2024-04-29T20:21:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.858+0000 7f1733fff640 10 client.4131.objecter ms_dispatch 0x7f174c000b90 osd_map(4..4 src has 1..4) 2024-04-29T20:21:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.858+0000 7f1733fff640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-04-29T20:21:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.858+0000 7f1733fff640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-04-29T20:21:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.859+0000 7f1733fff640 20 client.4131.objecter dump_active .. 0 homeless 2024-04-29T20:21:56.873 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:56.872+0000 7f06d7015040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-29T20:21:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.980+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f174003ad50 con 0x7f174c0b2230 2024-04-29T20:21:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.980+0000 7f1733fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:56.980+0000 7f1733fff640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:57.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.051+0000 7f1751bc7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2d380b76-312c-43ca-8301-7ec705d9fca3", "id": 2}] 2024-04-29T20:21:57.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.051+0000 7f1751bc7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2d380b76-312c-43ca-8301-7ec705d9fca3", "id": 2}] 2024-04-29T20:21:57.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.051+0000 7f1751bc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.053 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.051+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2d380b76-312c-43ca-8301-7ec705d9fca3", "id": 2} v 0) -- 0x7f174c071f70 con 0x7f174c0b2230 2024-04-29T20:21:57.069 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.068+0000 7f33fdc21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-29T20:21:57.121 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.120+0000 7f1733fff640 1 -- 172.21.15.167:0/1553604309 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d380b76-312c-43ca-8301-7ec705d9fca3", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f1740039430 con 0x7f174c0b2230 2024-04-29T20:21:57.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.120+0000 7f1733fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2d380b76-312c-43ca-8301-7ec705d9fca3", "id": 2}] 2024-04-29T20:21:57.122 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.120+0000 7f1733fff640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:57.122 INFO:teuthology.orchestra.run.smithi167.stdout:2 2024-04-29T20:21:57.124 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 10 librados: watch_flush enter 2024-04-29T20:21:57.124 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 10 librados: watch_flush exit 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 20 client.4131.objecter shutdown clearing up homeless session... 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 10 client.4131.objecter successfully canceled tick 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 10 monclient: shutdown 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0b2230 msgr2=0x7f174c070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0b2230 0x7f174c070a90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1740033000 tx=0x7f17400046c0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.125 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 shutdown_connections 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f174c0b5c60 0x7f174c1e05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f174c0b2230 0x7f174c070a90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 --2- 172.21.15.167:0/1553604309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f174c0aa460 0x7f174c070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 >> 172.21.15.167:0/1553604309 conn(0x7f174c06c960 msgr2=0x7f174c0b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 shutdown_connections 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 -- 172.21.15.167:0/1553604309 wait complete. 2024-04-29T20:21:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.121+0000 7f1751bc7640 1 librados: shutdown 2024-04-29T20:21:57.127 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.122+0000 7f33fdc21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-29T20:21:57.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 2190b44c-dc16-4b94-9a4c-931c87d0e875 3 2024-04-29T20:21:57.155 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:57.154+0000 7f06d7015040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-29T20:21:57.210 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:57.208+0000 7f06d7015040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-29T20:21:57.230 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.228+0000 7fe889b6c640 10 monclient: get_monmap_and_config 2024-04-29T20:21:57.230 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.229+0000 7fe889b6c640 10 monclient: build_initial_monmap 2024-04-29T20:21:57.230 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.229+0000 7fe889b6c640 10 monclient: monmap: 2024-04-29T20:21:57.230 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:57.230 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.230 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:57.229737+0000 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:57.229737+0000 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:57.231 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.234 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.229+0000 7fe889b6c640 1 Processor -- start 2024-04-29T20:21:57.234 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 1 -- start start 2024-04-29T20:21:57.234 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient: init 2024-04-29T20:21:57.234 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:57.234 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-29T20:21:57.234 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840b2230 0x7fe884146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fe8840b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fe8840aa460 0x7fe8840b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fe8840aa460 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8840b5c60 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe884146eb0 con 0x7fe8840b5c60 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fe8840ac1f0 con 0x7fe8840aa460 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fe8840ac4d0 con 0x7fe8840b2230 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.230+0000 7fe889b6c640 10 monclient(hunting): authenticate will time out at 1407.620146s 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe883fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fe8840b5c60 0x7fe8840b37b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57202/0 (socket says 172.21.15.167:57202) 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe883fff640 1 -- 172.21.15.167:0/1876668650 learned_addr learned my addr 172.21.15.167:0/1876668650 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe87bfff640 1 --2- 172.21.15.167:0/1876668650 >> v2:172.21.15.64:3300/0 conn(0x7fe8840aa460 0x7fe8840b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe8837fe640 1 --2- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840b2230 0x7fe884146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe87bfff640 10 monclient(hunting): get_auth_request con 0x7fe8840aa460 auth_method 0 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe87bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe87bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe8837fe640 10 monclient(hunting): get_auth_request con 0x7fe8840b2230 auth_method 0 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe8837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe8837fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.231+0000 7fe8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.232+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.232+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.232+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.232+0000 7fe8837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.232+0000 7fe8837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.232+0000 7fe8837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.238 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.233+0000 7fe882ffd640 1 -- 172.21.15.167:0/1876668650 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714080226 0 0) 0x7fe884146eb0 con 0x7fe8840b5c60 2024-04-29T20:21:57.238 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.233+0000 7fe882ffd640 10 monclient(hunting): _negotiate 2024-04-29T20:21:57.238 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.233+0000 7fe882ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.233+0000 7fe882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.233+0000 7fe882ffd640 10 monclient(hunting): my global_id is 4129 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.233+0000 7fe882ffd640 1 -- 172.21.15.167:0/1876668650 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe870003520 con 0x7fe8840b5c60 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 1 -- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840b2230 msgr2=0x7fe884146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 1 --2- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840b2230 0x7fe884146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 1 -- 172.21.15.167:0/1876668650 >> v1:172.21.15.167:6789/0 conn(0x7fe8840b5c60 legacy=0x7fe8840b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 1 monclient: found mon.noname-a 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.234+0000 7fe87bfff640 1 -- 172.21.15.167:0/1876668650 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8840b3440 con 0x7fe8840aa460 2024-04-29T20:21:57.239 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.236+0000 7fe87bfff640 1 --2- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840aa460 0x7fe8840b2f00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0097a0 tx=0x7fe86c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d7d17ed6979747 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:57.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 1 -- 172.21.15.167:0/1876668650 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0315a0 con 0x7fe8840aa460 2024-04-29T20:21:57.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: dump: 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: _finish_auth 0 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 1 -- 172.21.15.167:0/1876668650 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c037070 con 0x7fe8840aa460 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe889b6c640 5 monclient: authenticate success, global_id 4134 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe889b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe889b6c640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe889b6c640 4 set_mon_vals no callback set 2024-04-29T20:21:57.244 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 1 -- 172.21.15.167:0/1876668650 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c031960 con 0x7fe8840aa460 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.239+0000 7fe882ffd640 10 monclient: dump: 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 10 monclient: shutdown 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 -- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840aa460 msgr2=0x7fe8840b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 --2- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840aa460 0x7fe8840b2f00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0097a0 tx=0x7fe86c02b840 comp rx=0 tx=0).stop 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 -- 172.21.15.167:0/1876668650 shutdown_connections 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 --2- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840aa460 0x7fe8840b2f00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 --2- 172.21.15.167:0/1876668650 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840b2230 0x7fe884146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 -- 172.21.15.167:0/1876668650 >> 172.21.15.167:0/1876668650 conn(0x7fe88406c960 msgr2=0x7fe88406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 -- 172.21.15.167:0/1876668650 shutdown_connections 2024-04-29T20:21:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.240+0000 7fe889b6c640 1 -- 172.21.15.167:0/1876668650 wait complete. 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 10 monclient: build_initial_monmap 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 10 monclient: monmap: 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 1 librados: starting msgr at 2024-04-29T20:21:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 1 librados: starting objecter 2024-04-29T20:21:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 1 Processor -- start 2024-04-29T20:21:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 1 -- start start 2024-04-29T20:21:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 1 librados: setting wanted keys 2024-04-29T20:21:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 1 librados: calling monclient init 2024-04-29T20:21:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.241+0000 7fe889b6c640 10 monclient: init 2024-04-29T20:21:57.251 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840aa460 0x7fe8841435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): picked mon.noname-c con 0x7fe8840aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840b2230 0x7fe8840ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fe8840b2230 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8840b5c60 0x7fe88413bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fe8840b5c60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe8840b6250 con 0x7fe8840b2230 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe8840b60d0 con 0x7fe8840b5c60 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fe8840b2610 con 0x7fe8840aa460 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe889b6c640 10 monclient(hunting): authenticate will time out at 1407.631677s 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe883fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8840b5c60 0x7fe88413bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.242+0000 7fe87bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840b2230 0x7fe8840ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe8837fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840aa460 0x7fe8841435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8840b5c60 0x7fe88413bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59510/0 (socket says 172.21.15.167:59510) 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 1 -- 172.21.15.167:0/2391650089 learned_addr learned my addr 172.21.15.167:0/2391650089 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): get_auth_request con 0x7fe8840b5c60 auth_method 0 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): get_auth_request con 0x7fe8840b2230 auth_method 0 2024-04-29T20:21:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe8837fe640 10 monclient(hunting): get_auth_request con 0x7fe8840aa460 auth_method 0 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe8837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe8837fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.243+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.244+0000 7fe8837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.244+0000 7fe8837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.244+0000 7fe8837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 10 monclient(hunting): handle_auth_done global_id 4137 payload 824 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 1 -- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840aa460 msgr2=0x7fe8841435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840aa460 0x7fe8841435f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 1 -- 172.21.15.167:0/2391650089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8840b5c60 msgr2=0x7fe88413bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8840b5c60 0x7fe88413bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 1 monclient: found mon.noname-a 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.246+0000 7fe87bfff640 1 -- 172.21.15.167:0/2391650089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8840af550 con 0x7fe8840b2230 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.248+0000 7fe87bfff640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840b2230 0x7fe8840ae4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe86c033000 tx=0x7fe86c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.248+0000 7fe880ff9640 10 client.?.objecter ms_handle_connect 0x7fe8840b2230 2024-04-29T20:21:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.248+0000 7fe880ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:57.256 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.250+0000 7f33fdc21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-29T20:21:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c032630 con 0x7fe8840b2230 2024-04-29T20:21:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:21:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: dump: 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: _finish_auth 0 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:21:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0327f0 con 0x7fe8840b2230 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c032be0 con 0x7fe8840b2230 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe880ff9640 10 monclient: dump: 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 5 monclient: authenticate success, global_id 4137 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 10 monclient: _renew_subs 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8840aeb10 con 0x7fe8840b2230 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe881ffb640 4 set_mon_vals no callback set 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 10 monclient: _renew_subs 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88413c640 con 0x7fe8840b2230 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.252+0000 7fe889b6c640 1 librados: init done 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.253+0000 7fe889b6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.253+0000 7fe889b6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.253+0000 7fe889b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.253+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe88404fce0 con 0x7fe8840b2230 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.255+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe86c04e7c0 con 0x7fe8840b2230 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.255+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fe86c02f030 con 0x7fe8840b2230 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.255+0000 7fe880ff9640 10 client.4137.objecter ms_dispatch 0x7fe884000b90 osd_map(5..5 src has 1..5) 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.255+0000 7fe880ff9640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.255+0000 7fe880ff9640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-04-29T20:21:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.255+0000 7fe880ff9640 20 client.4137.objecter dump_active .. 0 homeless 2024-04-29T20:21:57.338 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:57.337+0000 7f06d7015040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-29T20:21:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.364+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe86c040070 con 0x7fe8840b2230 2024-04-29T20:21:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.364+0000 7fe880ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.364+0000 7fe880ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:57.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.437+0000 7fe889b6c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2190b44c-dc16-4b94-9a4c-931c87d0e875", "id": 3}] 2024-04-29T20:21:57.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.437+0000 7fe889b6c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2190b44c-dc16-4b94-9a4c-931c87d0e875", "id": 3}] 2024-04-29T20:21:57.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.437+0000 7fe889b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.439 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.437+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2190b44c-dc16-4b94-9a4c-931c87d0e875", "id": 3} v 0) -- 0x7fe8840b0440 con 0x7fe8840b2230 2024-04-29T20:21:57.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.475+0000 7fe880ff9640 1 -- 172.21.15.167:0/2391650089 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2190b44c-dc16-4b94-9a4c-931c87d0e875", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fe86c032d80 con 0x7fe8840b2230 2024-04-29T20:21:57.476 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.475+0000 7fe880ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2190b44c-dc16-4b94-9a4c-931c87d0e875", "id": 3}] 2024-04-29T20:21:57.477 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.475+0000 7fe880ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:57.477 INFO:teuthology.orchestra.run.smithi167.stdout:3 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 10 librados: watch_flush enter 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 10 librados: watch_flush exit 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 20 client.4137.objecter shutdown clearing up homeless session... 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 10 client.4137.objecter successfully canceled tick 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 10 monclient: shutdown 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840b2230 msgr2=0x7fe8840ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840b2230 0x7fe8840ae4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe86c033000 tx=0x7fe86c0046c0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 shutdown_connections 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8840b5c60 0x7fe88413bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8840b2230 0x7fe8840ae4a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 --2- 172.21.15.167:0/2391650089 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe8840aa460 0x7fe8841435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 >> 172.21.15.167:0/2391650089 conn(0x7fe88406c960 msgr2=0x7fe884146950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 shutdown_connections 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 -- 172.21.15.167:0/2391650089 wait complete. 2024-04-29T20:21:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.476+0000 7fe889b6c640 1 librados: shutdown 2024-04-29T20:21:57.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new f20c5825-9198-44fb-9b31-7de381a3310c 4 2024-04-29T20:21:57.543 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.542+0000 7f33fdc21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-29T20:21:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.583+0000 7ff10e2b1640 10 monclient: get_monmap_and_config 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.583+0000 7ff10e2b1640 10 monclient: build_initial_monmap 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.583+0000 7ff10e2b1640 10 monclient: monmap: 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:57.584286+0000 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:57.584286+0000 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:57.585 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 1 Processor -- start 2024-04-29T20:21:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 1 -- start start 2024-04-29T20:21:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient: init 2024-04-29T20:21:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1080b2230 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1080aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff1080b5c60 0x7ff10814ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient(hunting): picked mon.noname-a con 0x7ff1080b5c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080ac1f0 con 0x7ff1080b2230 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080ac4d0 con 0x7ff1080aa460 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7ff1080aa8d0 con 0x7ff1080b5c60 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.584+0000 7ff10e2b1640 10 monclient(hunting): authenticate will time out at 1407.974321s 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff10c84e640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff1080b5c60 0x7ff10814ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff1077fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7ff1080aa460 0x7ff1080b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57206/0 (socket says 172.21.15.167:57206) 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff1077fe640 1 -- 172.21.15.167:0/4087410875 learned_addr learned my addr 172.21.15.167:0/4087410875 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff10c84e640 10 monclient(hunting): get_auth_request con 0x7ff1080b5c60 auth_method 0 2024-04-29T20:21:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff10c84e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff10c84e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.585+0000 7ff10c84e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.586+0000 7ff10c84e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.586+0000 7ff10c84e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.586+0000 7ff10c84e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff106ffd640 1 -- 172.21.15.167:0/4087410875 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895075880 0 0) 0x7ff1080ac4d0 con 0x7ff1080aa460 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff106ffd640 10 monclient(hunting): _negotiate 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff106ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff106ffd640 10 monclient(hunting): my global_id is 4135 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff106ffd640 1 -- 172.21.15.167:0/4087410875 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f0003520 con 0x7ff1080aa460 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 1 -- 172.21.15.167:0/4087410875 >> v1:172.21.15.167:6789/0 conn(0x7ff1080aa460 legacy=0x7ff1080b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 1 -- 172.21.15.167:0/4087410875 >> v1:172.21.15.64:6789/0 conn(0x7ff1080b2230 legacy=0x7ff108146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 10 monclient: _finish_hunting 0 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 1 monclient: found mon.noname-a 2024-04-29T20:21:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:21:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.588+0000 7ff10c84e640 1 -- 172.21.15.167:0/4087410875 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1080b2610 con 0x7ff1080b5c60 2024-04-29T20:21:57.593 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.592+0000 7ff10c84e640 1 --2- 172.21.15.167:0/4087410875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff10814ecf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff100009d90 tx=0x7ff10002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8bb10af1fa2c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:57.595 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.594+0000 7f33fdc21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-29T20:21:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 1 -- 172.21.15.167:0/4087410875 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100039070 con 0x7ff1080b5c60 2024-04-29T20:21:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:21:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: dump: 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: _finish_auth 0 2024-04-29T20:21:57.598 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 1 -- 172.21.15.167:0/4087410875 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10002cc60 con 0x7ff1080b5c60 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 1 -- 172.21.15.167:0/4087410875 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100034660 con 0x7ff1080b5c60 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff10e2b1640 5 monclient: authenticate success, global_id 4140 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff10e2b1640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff10e2b1640 4 set_mon_vals no callback set 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.596+0000 7ff106ffd640 10 monclient: dump: 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.600 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 10 monclient: shutdown 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 -- 172.21.15.167:0/4087410875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 msgr2=0x7ff10814ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 --2- 172.21.15.167:0/4087410875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff10814ecf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff100009d90 tx=0x7ff10002beb0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 -- 172.21.15.167:0/4087410875 shutdown_connections 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 --2- 172.21.15.167:0/4087410875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff10814ecf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 -- 172.21.15.167:0/4087410875 >> 172.21.15.167:0/4087410875 conn(0x7ff10806c960 msgr2=0x7ff10806cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 -- 172.21.15.167:0/4087410875 shutdown_connections 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 1 -- 172.21.15.167:0/4087410875 wait complete. 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 10 monclient: build_initial_monmap 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.597+0000 7ff10e2b1640 10 monclient: monmap: 2024-04-29T20:21:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 librados: starting msgr at 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 librados: starting objecter 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 Processor -- start 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 -- start start 2024-04-29T20:21:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 librados: setting wanted keys 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 librados: calling monclient init 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient: init 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1080aa460 0x7ff10813fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): picked mon.noname-c con 0x7ff1080aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1080b2230 0x7ff1080b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): picked mon.noname-b con 0x7ff1080b2230 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff1080aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): picked mon.noname-a con 0x7ff1080b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff10814f3b0 con 0x7ff1080b5c60 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff10814f230 con 0x7ff1080b2230 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff10814f530 con 0x7ff1080aa460 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.598+0000 7ff10e2b1640 10 monclient(hunting): authenticate will time out at 1407.988316s 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff1077fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1080b2230 0x7ff1080b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff1080aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff107fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1080aa460 0x7ff10813fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff1080aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.167:51388/0 (socket says 172.21.15.167:51388) 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 1 -- 172.21.15.167:0/3943207636 learned_addr learned my addr 172.21.15.167:0/3943207636 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 10 monclient(hunting): get_auth_request con 0x7ff1080b5c60 auth_method 0 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff10c84e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff1077fe640 10 monclient(hunting): get_auth_request con 0x7ff1080b2230 auth_method 0 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff1077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff1077fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.599+0000 7ff1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): get_auth_request con 0x7ff1080aa460 auth_method 0 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff10c84e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff10c84e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff10c84e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff1077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff1077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff1077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.608 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.600+0000 7ff107fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 1 -- 172.21.15.167:0/3943207636 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1080aa460 msgr2=0x7ff10813fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1080aa460 0x7ff10813fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 1 -- 172.21.15.167:0/3943207636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 msgr2=0x7ff1080aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff1080aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 10 monclient: _finish_hunting 0 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 1 monclient: found mon.noname-b 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.603+0000 7ff1077fe640 1 -- 172.21.15.167:0/3943207636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1080ae9e0 con 0x7ff1080b2230 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.605+0000 7ff1077fe640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1080b2230 0x7ff1080b0a90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c400 tx=0x7ff0fc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.605+0000 7ff104ff9640 10 client.?.objecter ms_handle_connect 0x7ff1080b2230 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.605+0000 7ff104ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc017070 con 0x7ff1080b2230 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 10 monclient: dump: 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 10 monclient: _finish_auth 0 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff104ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 5 monclient: authenticate success, global_id 4138 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 10 monclient: _renew_subs 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1080b00b0 con 0x7ff1080b2230 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 10 monclient: _renew_subs 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1080b0670 con 0x7ff1080b2230 2024-04-29T20:21:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.607+0000 7ff10e2b1640 1 librados: init done 2024-04-29T20:21:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.608+0000 7ff10e2b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.608+0000 7ff10e2b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.608+0000 7ff10e2b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.608+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10804fce0 con 0x7ff1080b2230 2024-04-29T20:21:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0fc005ce0 con 0x7ff1080b2230 2024-04-29T20:21:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff104ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc021060 con 0x7ff1080b2230 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff104ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff104ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff104ff9640 10 monclient: dump: 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.609+0000 7ff1067fc640 4 set_mon_vals no callback set 2024-04-29T20:21:57.641 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:57.640+0000 7f06d7015040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-29T20:21:57.691 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:57.689+0000 7f06d7015040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff0fc01a070 con 0x7ff1080b2230 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff0fc012910 con 0x7ff1080b2230 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 10 client.4138.objecter ms_dispatch 0x7ff108000b90 osd_map(6..6 src has 1..6) 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff0fc012b80 con 0x7ff1080b2230 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.712+0000 7ff104ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:57.757 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.756+0000 7f33fdc21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-29T20:21:57.786 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.785+0000 7ff10e2b1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f20c5825-9198-44fb-9b31-7de381a3310c", "id": 4}] 2024-04-29T20:21:57.786 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.785+0000 7ff10e2b1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f20c5825-9198-44fb-9b31-7de381a3310c", "id": 4}] 2024-04-29T20:21:57.786 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.785+0000 7ff10e2b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.786 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.785+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f20c5825-9198-44fb-9b31-7de381a3310c", "id": 4} v 0) -- 0x7ff1080aebb0 con 0x7ff1080b2230 2024-04-29T20:21:57.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff104ff9640 1 -- 172.21.15.167:0/3943207636 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f20c5825-9198-44fb-9b31-7de381a3310c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff0fc012d60 con 0x7ff1080b2230 2024-04-29T20:21:57.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff104ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f20c5825-9198-44fb-9b31-7de381a3310c", "id": 4}] 2024-04-29T20:21:57.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff104ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:57.859 INFO:teuthology.orchestra.run.smithi167.stdout:4 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff10e2b1640 10 librados: watch_flush enter 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff10e2b1640 10 librados: watch_flush exit 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff10e2b1640 20 client.4138.objecter shutdown clearing up homeless session... 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.858+0000 7ff10e2b1640 10 client.4138.objecter successfully canceled tick 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 10 monclient: shutdown 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1080b2230 msgr2=0x7ff1080b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1080b2230 0x7ff1080b0a90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c400 tx=0x7ff0fc00c8c0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 shutdown_connections 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1080b5c60 0x7ff1080aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1080b2230 0x7ff1080b0a90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.861 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 --2- 172.21.15.167:0/3943207636 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1080aa460 0x7ff10813fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 >> 172.21.15.167:0/3943207636 conn(0x7ff10806c960 msgr2=0x7ff1080b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:57.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 shutdown_connections 2024-04-29T20:21:57.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 -- 172.21.15.167:0/3943207636 wait complete. 2024-04-29T20:21:57.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.859+0000 7ff10e2b1640 1 librados: shutdown 2024-04-29T20:21:57.862 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:57.860+0000 7f06d7015040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-29T20:21:57.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new a4937fcb-dbd8-4cc1-ac32-ddf147f0469b 5 2024-04-29T20:21:57.901 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.900+0000 7f33fdc21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-29T20:21:57.953 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:57.952+0000 7f33fdc21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.967+0000 7fbf924a8640 10 monclient: get_monmap_and_config 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.967+0000 7fbf924a8640 10 monclient: build_initial_monmap 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.967+0000 7fbf924a8640 10 monclient: monmap: 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:57.967745+0000 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:57.967745+0000 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.968 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:57.969 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:57.969 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:57.969 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:57.969 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:57.969 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.970 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.967+0000 7fbf924a8640 1 Processor -- start 2024-04-29T20:21:57.970 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.967+0000 7fbf924a8640 1 -- start start 2024-04-29T20:21:57.970 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.967+0000 7fbf924a8640 10 monclient: init 2024-04-29T20:21:57.970 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b2230 0x7fbf8c146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8c0b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf8c0aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf8c0b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c146eb0 con 0x7fbf8c0b5c60 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c0ac1f0 con 0x7fbf8c0aa460 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c0ac4d0 con 0x7fbf8c0b2230 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.968+0000 7fbf924a8640 10 monclient(hunting): authenticate will time out at 1408.357767s 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8b7fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fbf8c0aa460 0x7fbf8c0b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57212/0 (socket says 172.21.15.167:57212) 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8b7fe640 1 -- 172.21.15.167:0/2719608423 learned_addr learned my addr 172.21.15.167:0/2719608423 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8bfff640 1 --2- 172.21.15.167:0/2719608423 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b2230 0x7fbf8c146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.972 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8bfff640 10 monclient(hunting): get_auth_request con 0x7fbf8c0b2230 auth_method 0 2024-04-29T20:21:57.972 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.972 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.972 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.969+0000 7fbf8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.972 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.970+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.970+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.970+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 811156786 0 0) 0x7fbf8c0ac1f0 con 0x7fbf8c0aa460 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): _negotiate 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): my global_id is 4141 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf74003520 con 0x7fbf8c0aa460 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3681115252 0 0) 0x7fbf8c146eb0 con 0x7fbf8c0b5c60 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): _negotiate 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.973 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 10 monclient(hunting): my global_id is 4146 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.971+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf8c0ac1f0 con 0x7fbf8c0b5c60 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 1 -- 172.21.15.167:0/2719608423 >> v1:172.21.15.167:6789/0 conn(0x7fbf8c0aa460 legacy=0x7fbf8c0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 1 -- 172.21.15.167:0/2719608423 >> v1:172.21.15.64:6789/0 conn(0x7fbf8c0b5c60 legacy=0x7fbf8c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:57.974 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 1 monclient: found mon.noname-b 2024-04-29T20:21:57.975 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:57.975 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:21:57.975 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.972+0000 7fbf8bfff640 1 -- 172.21.15.167:0/2719608423 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c0aa8a0 con 0x7fbf8c0b2230 2024-04-29T20:21:57.976 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.975+0000 7fbf8bfff640 1 --2- 172.21.15.167:0/2719608423 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b2230 0x7fbf8c146970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf800097d0 tx=0x7fbf8002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4edb1f4aec93ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80035020 con 0x7fbf8c0b2230 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: dump: 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.981 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: _finish_auth 0 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80031890 con 0x7fbf8c0b2230 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 1 -- 172.21.15.167:0/2719608423 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80031c60 con 0x7fbf8c0b2230 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 5 monclient: authenticate success, global_id 4127 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:57.982 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 4 set_mon_vals no callback set 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf8affd640 10 monclient: dump: 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.983 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 10 monclient: shutdown 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 1 -- 172.21.15.167:0/2719608423 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b2230 msgr2=0x7fbf8c146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 1 --2- 172.21.15.167:0/2719608423 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b2230 0x7fbf8c146970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf800097d0 tx=0x7fbf8002b930 comp rx=0 tx=0).stop 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 1 -- 172.21.15.167:0/2719608423 shutdown_connections 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 1 --2- 172.21.15.167:0/2719608423 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b2230 0x7fbf8c146970 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 1 -- 172.21.15.167:0/2719608423 >> 172.21.15.167:0/2719608423 conn(0x7fbf8c06c960 msgr2=0x7fbf8c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.980+0000 7fbf924a8640 1 -- 172.21.15.167:0/2719608423 shutdown_connections 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.981+0000 7fbf924a8640 1 -- 172.21.15.167:0/2719608423 wait complete. 2024-04-29T20:21:57.987 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.981+0000 7fbf924a8640 10 monclient: build_initial_monmap 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.981+0000 7fbf924a8640 10 monclient: monmap: 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.981+0000 7fbf924a8640 1 librados: starting msgr at 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.981+0000 7fbf924a8640 1 librados: starting objecter 2024-04-29T20:21:57.988 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 Processor -- start 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 -- start start 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 librados: setting wanted keys 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 librados: calling monclient init 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient: init 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 0x7fbf8c143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8c0aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf8c0b2230 0x7fbf8c0ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): picked mon.noname-a con 0x7fbf8c0b2230 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b5c60 0x7fbf8c13fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): picked mon.noname-c con 0x7fbf8c0b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c0b2790 con 0x7fbf8c0b2230 2024-04-29T20:21:57.989 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c0b2610 con 0x7fbf8c0aa460 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c14f400 con 0x7fbf8c0b5c60 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.982+0000 7fbf924a8640 10 monclient(hunting): authenticate will time out at 1408.372184s 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 0x7fbf8c143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf90a45640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b5c60 0x7fbf8c13fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf8c0b2230 0x7fbf8c0ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 0x7fbf8c143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59520/0 (socket says 172.21.15.167:59520) 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 1 -- 172.21.15.167:0/272229431 learned_addr learned my addr 172.21.15.167:0/272229431 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 10 monclient(hunting): get_auth_request con 0x7fbf8c0aa460 auth_method 0 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.990 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8b7fe640 10 monclient(hunting): get_auth_request con 0x7fbf8c0b2230 auth_method 0 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf90a45640 10 monclient(hunting): get_auth_request con 0x7fbf8c0b5c60 auth_method 0 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf90a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf90a45640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.983+0000 7fbf90a45640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.991 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf90a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:57.992 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf90a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:57.992 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.984+0000 7fbf90a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:57.992 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.986+0000 7fbf8bfff640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 1 -- 172.21.15.167:0/272229431 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b5c60 msgr2=0x7fbf8c13fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b5c60 0x7fbf8c13fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 1 -- 172.21.15.167:0/272229431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf8c0b2230 msgr2=0x7fbf8c0ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf8c0b2230 0x7fbf8c0ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 1 monclient: found mon.noname-b 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.987+0000 7fbf8bfff640 1 -- 172.21.15.167:0/272229431 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c0acb50 con 0x7fbf8c0aa460 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.989+0000 7fbf8bfff640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 0x7fbf8c143cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf800098f0 tx=0x7fbf800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.989+0000 7fbf88ff9640 10 client.?.objecter ms_handle_connect 0x7fbf8c0aa460 2024-04-29T20:21:57.993 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.989+0000 7fbf88ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80035040 con 0x7fbf8c0aa460 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: dump: 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.995 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: _finish_auth 0 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80030ca0 con 0x7fbf8c0aa460 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 5 monclient: authenticate success, global_id 4144 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 10 monclient: _renew_subs 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c13f970 con 0x7fbf8c0aa460 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf800466c0 con 0x7fbf8c0aa460 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf88ff9640 10 monclient: dump: 2024-04-29T20:21:57.996 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf8a7fc640 4 set_mon_vals no callback set 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 10 monclient: _renew_subs 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c0ad0a0 con 0x7fbf8c0aa460 2024-04-29T20:21:57.997 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.992+0000 7fbf924a8640 1 librados: init done 2024-04-29T20:21:57.998 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.993+0000 7fbf924a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.998 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.993+0000 7fbf924a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:57.998 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.993+0000 7fbf924a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:57.998 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:57.993+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c04fce0 con 0x7fbf8c0aa460 2024-04-29T20:21:58.014 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:58.013+0000 7f06d7015040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-29T20:21:58.028 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:58.027+0000 7f33fdc21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-29T20:21:58.066 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:58.064+0000 7f06d7015040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-29T20:21:58.074 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:58.073+0000 7f33fdc21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-29T20:21:58.097 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbf80038020 con 0x7fbf8c0aa460 2024-04-29T20:21:58.097 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fbf8003bcb0 con 0x7fbf8c0aa460 2024-04-29T20:21:58.097 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 10 client.4144.objecter ms_dispatch 0x7fbf8c000b90 osd_map(7..7 src has 1..7) 2024-04-29T20:21:58.097 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-04-29T20:21:58.097 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-04-29T20:21:58.097 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 20 client.4144.objecter dump_active .. 0 homeless 2024-04-29T20:21:58.098 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fbf8002d040 con 0x7fbf8c0aa460 2024-04-29T20:21:58.098 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.098 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.095+0000 7fbf88ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:58.139 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:58.138+0000 7f06d7015040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-29T20:21:58.168 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.167+0000 7fbf924a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a4937fcb-dbd8-4cc1-ac32-ddf147f0469b", "id": 5}] 2024-04-29T20:21:58.169 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.167+0000 7fbf924a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a4937fcb-dbd8-4cc1-ac32-ddf147f0469b", "id": 5}] 2024-04-29T20:21:58.169 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.167+0000 7fbf924a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.169 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.167+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a4937fcb-dbd8-4cc1-ac32-ddf147f0469b", "id": 5} v 0) -- 0x7fbf8c0b7300 con 0x7fbf8c0aa460 2024-04-29T20:21:58.186 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:58.185+0000 7f06d7015040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-29T20:21:58.230 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.228+0000 7fbf88ff9640 1 -- 172.21.15.167:0/272229431 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4937fcb-dbd8-4cc1-ac32-ddf147f0469b", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fbf8003b6f0 con 0x7fbf8c0aa460 2024-04-29T20:21:58.230 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.228+0000 7fbf88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a4937fcb-dbd8-4cc1-ac32-ddf147f0469b", "id": 5}] 2024-04-29T20:21:58.230 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.228+0000 7fbf88ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:58.230 INFO:teuthology.orchestra.run.smithi167.stdout:5 2024-04-29T20:21:58.232 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 10 librados: watch_flush enter 2024-04-29T20:21:58.232 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 10 librados: watch_flush exit 2024-04-29T20:21:58.232 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 20 client.4144.objecter shutdown clearing up homeless session... 2024-04-29T20:21:58.232 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 10 client.4144.objecter successfully canceled tick 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 10 monclient: shutdown 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 msgr2=0x7fbf8c143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 0x7fbf8c143cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf800098f0 tx=0x7fbf800046c0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 shutdown_connections 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fbf8c0b5c60 0x7fbf8c13fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf8c0b2230 0x7fbf8c0ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 --2- 172.21.15.167:0/272229431 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf8c0aa460 0x7fbf8c143cc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.229+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 >> 172.21.15.167:0/272229431 conn(0x7fbf8c06c960 msgr2=0x7fbf8c06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.230+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 shutdown_connections 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.230+0000 7fbf924a8640 1 -- 172.21.15.167:0/272229431 wait complete. 2024-04-29T20:21:58.233 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.230+0000 7fbf924a8640 1 librados: shutdown 2024-04-29T20:21:58.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new ac61b59e-81b4-47cd-a33c-aa3d02142b9c 6 2024-04-29T20:21:58.382 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.381+0000 7fca4959e640 10 monclient: get_monmap_and_config 2024-04-29T20:21:58.382 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.381+0000 7fca4959e640 10 monclient: build_initial_monmap 2024-04-29T20:21:58.382 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.381+0000 7fca4959e640 10 monclient: monmap: 2024-04-29T20:21:58.382 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:58.382 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:58.381909+0000 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:58.381909+0000 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:58.383 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.381+0000 7fca4959e640 1 Processor -- start 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.381+0000 7fca4959e640 1 -- start start 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.381+0000 7fca4959e640 10 monclient: init 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca44143c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): picked mon.noname-b con 0x7fca4406c110 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fca44144160 0x7fca440af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): picked mon.noname-c con 0x7fca44144160 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fca440afcc0 0x7fca440b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): picked mon.noname-a con 0x7fca440afcc0 addr v2:172.21.15.64:3300/0 2024-04-29T20:21:58.386 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fca4406d530 con 0x7fca440afcc0 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fca4406c4f0 con 0x7fca44144160 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fca440af080 con 0x7fca4406c110 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.382+0000 7fca4959e640 10 monclient(hunting): authenticate will time out at 1408.771950s 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fca44144160 0x7fca440af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca48d9d640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fca440afcc0 0x7fca440b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca43fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca44143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fca44144160 0x7fca440af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59528/0 (socket says 172.21.15.167:59528) 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 1 -- 172.21.15.167:0/3762834549 learned_addr learned my addr 172.21.15.167:0/3762834549 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 10 monclient(hunting): get_auth_request con 0x7fca44144160 auth_method 0 2024-04-29T20:21:58.387 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca48d9d640 10 monclient(hunting): get_auth_request con 0x7fca440afcc0 auth_method 0 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca48d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca48d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca48d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca43fff640 10 monclient(hunting): get_auth_request con 0x7fca4406c110 auth_method 0 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca43fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.383+0000 7fca43fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.384+0000 7fca437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.384+0000 7fca437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.388 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.384+0000 7fca437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.389 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.384+0000 7fca43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.389 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.384+0000 7fca43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.389 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.384+0000 7fca43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 1 -- 172.21.15.167:0/3762834549 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 msgr2=0x7fca44143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 1 --2- 172.21.15.167:0/3762834549 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca44143c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 1 -- 172.21.15.167:0/3762834549 >> v2:172.21.15.64:3300/0 conn(0x7fca440afcc0 msgr2=0x7fca440b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 1 --2- 172.21.15.167:0/3762834549 >> v2:172.21.15.64:3300/0 conn(0x7fca440afcc0 0x7fca440b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 10 monclient: _finish_hunting 0 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 1 monclient: found mon.noname-c 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.385+0000 7fca437fe640 1 -- 172.21.15.167:0/3762834549 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca440af360 con 0x7fca44144160 2024-04-29T20:21:58.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.387+0000 7fca437fe640 1 --2- 172.21.15.167:0/3762834549 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca440af780 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca440b7840 tx=0x7fca3802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a7325907186f90 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 1 -- 172.21.15.167:0/3762834549 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38035020 con 0x7fca44144160 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: dump: 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.393 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: _finish_auth 0 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 1 -- 172.21.15.167:0/3762834549 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca38031870 con 0x7fca44144160 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 1 -- 172.21.15.167:0/3762834549 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38031c40 con 0x7fca44144160 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca4959e640 5 monclient: authenticate success, global_id 4147 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca4959e640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca4959e640 4 set_mon_vals no callback set 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:58.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.391+0000 7fca42ffd640 10 monclient: dump: 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.395 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 10 monclient: shutdown 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 -- 172.21.15.167:0/3762834549 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 msgr2=0x7fca440af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 --2- 172.21.15.167:0/3762834549 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca440af780 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca440b7840 tx=0x7fca3802b910 comp rx=0 tx=0).stop 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 -- 172.21.15.167:0/3762834549 shutdown_connections 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 --2- 172.21.15.167:0/3762834549 >> v2:172.21.15.64:3300/0 conn(0x7fca440afcc0 0x7fca440b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 --2- 172.21.15.167:0/3762834549 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca440af780 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 --2- 172.21.15.167:0/3762834549 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca44143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 -- 172.21.15.167:0/3762834549 >> 172.21.15.167:0/3762834549 conn(0x7fca44068320 msgr2=0x7fca440b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 -- 172.21.15.167:0/3762834549 shutdown_connections 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.392+0000 7fca4959e640 1 -- 172.21.15.167:0/3762834549 wait complete. 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.393+0000 7fca4959e640 10 monclient: build_initial_monmap 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.393+0000 7fca4959e640 10 monclient: monmap: 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:58.400 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.393+0000 7fca4959e640 1 librados: starting msgr at 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.393+0000 7fca4959e640 1 librados: starting objecter 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.393+0000 7fca4959e640 1 Processor -- start 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 -- start start 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 librados: setting wanted keys 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 librados: calling monclient init 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient: init 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca441ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): picked mon.noname-c con 0x7fca4406c110 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:58.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca440afcc0 0x7fca441ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): picked mon.noname-a con 0x7fca440afcc0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca441dc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): picked mon.noname-b con 0x7fca44144160 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fca440b1120 con 0x7fca440afcc0 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fca440b0fa0 con 0x7fca44144160 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fca441445c0 con 0x7fca4406c110 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.394+0000 7fca4959e640 10 monclient(hunting): authenticate will time out at 1408.783975s 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.395+0000 7fca437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca440afcc0 0x7fca441ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.396+0000 7fca437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca440afcc0 0x7fca441ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.167:51422/0 (socket says 172.21.15.167:51422) 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.396+0000 7fca437fe640 1 -- 172.21.15.167:0/494169516 learned_addr learned my addr 172.21.15.167:0/494169516 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:58.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.396+0000 7fca437fe640 10 monclient(hunting): get_auth_request con 0x7fca440afcc0 auth_method 0 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.396+0000 7fca437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.396+0000 7fca437fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.396+0000 7fca437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca48d9d640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca441dc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca43fff640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca441ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca48d9d640 10 monclient(hunting): get_auth_request con 0x7fca44144160 auth_method 0 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca48d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca48d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.397+0000 7fca48d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): get_auth_request con 0x7fca4406c110 auth_method 0 2024-04-29T20:21:58.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca48d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca48d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca48d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.398+0000 7fca43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 1 -- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 msgr2=0x7fca441ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca441ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 1 -- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca440afcc0 msgr2=0x7fca441ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca440afcc0 0x7fca441ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 10 monclient: _finish_hunting 0 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 1 monclient: found mon.noname-b 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.401+0000 7fca48d9d640 1 -- 172.21.15.167:0/494169516 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca441cee50 con 0x7fca44144160 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.403+0000 7fca48d9d640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca441dc170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca3c007dc0 tx=0x7fca3c00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.403+0000 7fca40ff9640 10 client.?.objecter ms_handle_connect 0x7fca44144160 2024-04-29T20:21:58.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.403+0000 7fca40ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:58.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00edd0 con 0x7fca44144160 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: dump: 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: _finish_auth 0 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca3c003990 con 0x7fca44144160 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 5 monclient: authenticate success, global_id 4150 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c003d80 con 0x7fca44144160 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca40ff9640 10 monclient: dump: 2024-04-29T20:21:58.411 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 10 monclient: _renew_subs 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.412 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca441cf370 con 0x7fca44144160 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca427fc640 4 set_mon_vals no callback set 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 10 monclient: _renew_subs 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca441cf660 con 0x7fca44144160 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.407+0000 7fca4959e640 1 librados: init done 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.408+0000 7fca4959e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.408+0000 7fca4959e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.408+0000 7fca4959e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.408+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca44077010 con 0x7fca44144160 2024-04-29T20:21:58.512 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fca3c01b020 con 0x7fca44144160 2024-04-29T20:21:58.512 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fca3c014aa0 con 0x7fca44144160 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 10 client.4150.objecter ms_dispatch 0x7fca44000b90 osd_map(8..8 src has 1..8) 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 20 client.4150.objecter dump_active .. 0 homeless 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fca3c014dd0 con 0x7fca44144160 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.510+0000 7fca40ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:58.584 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:58.582+0000 7f33fdc21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-29T20:21:58.586 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.584+0000 7fca4959e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac61b59e-81b4-47cd-a33c-aa3d02142b9c", "id": 6}] 2024-04-29T20:21:58.586 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.584+0000 7fca4959e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac61b59e-81b4-47cd-a33c-aa3d02142b9c", "id": 6}] 2024-04-29T20:21:58.586 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.584+0000 7fca4959e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.586 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.584+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ac61b59e-81b4-47cd-a33c-aa3d02142b9c", "id": 6} v 0) -- 0x7fca441cf020 con 0x7fca44144160 2024-04-29T20:21:58.648 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.646+0000 7fca40ff9640 1 -- 172.21.15.167:0/494169516 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac61b59e-81b4-47cd-a33c-aa3d02142b9c", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fca3c0144e0 con 0x7fca44144160 2024-04-29T20:21:58.648 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.646+0000 7fca40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac61b59e-81b4-47cd-a33c-aa3d02142b9c", "id": 6}] 2024-04-29T20:21:58.648 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.646+0000 7fca40ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:58.648 INFO:teuthology.orchestra.run.smithi167.stdout:6 2024-04-29T20:21:58.649 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 10 librados: watch_flush enter 2024-04-29T20:21:58.649 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 10 librados: watch_flush exit 2024-04-29T20:21:58.649 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 20 client.4150.objecter shutdown clearing up homeless session... 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 10 client.4150.objecter successfully canceled tick 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 10 monclient: shutdown 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 msgr2=0x7fca441dc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca441dc170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca3c007dc0 tx=0x7fca3c00e4b0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 shutdown_connections 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca44144160 0x7fca441dc170 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fca440afcc0 0x7fca441ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 --2- 172.21.15.167:0/494169516 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fca4406c110 0x7fca441ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.647+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 >> 172.21.15.167:0/494169516 conn(0x7fca44068320 msgr2=0x7fca4406a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.648+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 shutdown_connections 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.648+0000 7fca4959e640 1 -- 172.21.15.167:0/494169516 wait complete. 2024-04-29T20:21:58.650 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.648+0000 7fca4959e640 1 librados: shutdown 2024-04-29T20:21:58.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new f4d72d02-c58f-4ff9-8b55-f6de73ba6460 7 2024-04-29T20:21:58.706 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:58.705+0000 7f06d7015040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-29T20:21:58.709 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:21:58.708+0000 7f33fdc21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-29T20:21:58.776 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.775+0000 7fe3e66f2640 10 monclient: get_monmap_and_config 2024-04-29T20:21:58.776 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.775+0000 7fe3e66f2640 10 monclient: build_initial_monmap 2024-04-29T20:21:58.776 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.775+0000 7fe3e66f2640 10 monclient: monmap: 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:58.776182+0000 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:58.776182+0000 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:58.777 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.779 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 1 Processor -- start 2024-04-29T20:21:58.779 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 1 -- start start 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient: init 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fe3e00b5c60 0x7fe3e0143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient(hunting): picked mon.noname-c con 0x7fe3e00b5c60 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3e00b6040 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:58.780 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3e00b2f00 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e01439b0 con 0x7fe3e00b2f00 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e00b27a0 con 0x7fe3e00b6040 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e00ac1f0 con 0x7fe3e00b5c60 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.776+0000 7fe3e66f2640 10 monclient(hunting): authenticate will time out at 1409.166246s 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.777+0000 7fe3dffff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fe3e00b5c60 0x7fe3e0143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.777+0000 7fe3df7fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fe3e00b6040 0x7fe3e00aa460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:57218/0 (socket says 172.21.15.167:57218) 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.777+0000 7fe3df7fe640 1 -- 172.21.15.167:0/261039001 learned_addr learned my addr 172.21.15.167:0/261039001 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): get_auth_request con 0x7fe3e00b5c60 auth_method 0 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.781 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.782 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.782 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.778+0000 7fe3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.782 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2024-04-29T20:21:58.782 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 1 -- 172.21.15.167:0/261039001 >> v1:172.21.15.167:6789/0 conn(0x7fe3e00b6040 legacy=0x7fe3e00aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.782 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 1 -- 172.21.15.167:0/261039001 >> v1:172.21.15.64:6789/0 conn(0x7fe3e00b2f00 legacy=0x7fe3e00b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:21:58.783 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:58.783 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 1 monclient: found mon.noname-c 2024-04-29T20:21:58.783 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:58.783 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.783 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.780+0000 7fe3dffff640 1 -- 172.21.15.167:0/261039001 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e00ac4d0 con 0x7fe3e00b5c60 2024-04-29T20:21:58.785 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.784+0000 7fe3dffff640 1 --2- 172.21.15.167:0/261039001 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b5c60 0x7fe3e0143470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe3d00097a0 tx=0x7fe3d002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bcd31efc9df0004 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 1 -- 172.21.15.167:0/261039001 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0038070 con 0x7fe3e00b5c60 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: dump: 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.791 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: _finish_auth 0 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 1 -- 172.21.15.167:0/261039001 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d0031700 con 0x7fe3e00b5c60 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 5 monclient: authenticate success, global_id 4153 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 4 set_mon_vals no callback set 2024-04-29T20:21:58.792 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 1 -- 172.21.15.167:0/261039001 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0031ad0 con 0x7fe3e00b5c60 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3deffd640 10 monclient: dump: 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.793 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 10 monclient: shutdown 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 1 -- 172.21.15.167:0/261039001 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b5c60 msgr2=0x7fe3e0143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 1 --2- 172.21.15.167:0/261039001 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b5c60 0x7fe3e0143470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe3d00097a0 tx=0x7fe3d002b7a0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 1 -- 172.21.15.167:0/261039001 shutdown_connections 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 1 --2- 172.21.15.167:0/261039001 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b5c60 0x7fe3e0143470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.789+0000 7fe3e66f2640 1 -- 172.21.15.167:0/261039001 >> 172.21.15.167:0/261039001 conn(0x7fe3e006c960 msgr2=0x7fe3e006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.790+0000 7fe3e66f2640 1 -- 172.21.15.167:0/261039001 shutdown_connections 2024-04-29T20:21:58.794 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.790+0000 7fe3e66f2640 1 -- 172.21.15.167:0/261039001 wait complete. 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.790+0000 7fe3e66f2640 10 monclient: build_initial_monmap 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.790+0000 7fe3e66f2640 10 monclient: monmap: 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.797 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 librados: starting msgr at 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 librados: starting objecter 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 Processor -- start 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 -- start start 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 librados: setting wanted keys 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 librados: calling monclient init 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 10 monclient: init 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:21:58.798 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 0x7fe3e00ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 10 monclient(hunting): picked mon.noname-b con 0x7fe3e00b2f00 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3e00b5c60 0x7fe3e00b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 10 monclient(hunting): picked mon.noname-a con 0x7fe3e00b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe3e01d98a0 0x7fe3e01dc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.791+0000 7fe3e66f2640 10 monclient(hunting): picked mon.noname-c con 0x7fe3e01d98a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e014eed0 con 0x7fe3e00b5c60 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e014ed50 con 0x7fe3e00b2f00 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e014f050 con 0x7fe3e01d98a0 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e66f2640 10 monclient(hunting): authenticate will time out at 1409.181394s 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 0x7fe3e00ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3e4c8f640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe3e01d98a0 0x7fe3e01dc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 0x7fe3e00ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:59562/0 (socket says 172.21.15.167:59562) 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 1 -- 172.21.15.167:0/2029079898 learned_addr learned my addr 172.21.15.167:0/2029079898 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 10 monclient(hunting): get_auth_request con 0x7fe3e00b2f00 auth_method 0 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.792+0000 7fe3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): get_auth_request con 0x7fe3e01d98a0 auth_method 0 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:58.800 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.793+0000 7fe3e4c8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.795+0000 7fe3dffff640 10 monclient(hunting): handle_auth_done global_id 4159 payload 824 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 1 -- 172.21.15.167:0/2029079898 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe3e01d98a0 msgr2=0x7fe3e01dc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe3e01d98a0 0x7fe3e01dc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 1 -- 172.21.15.167:0/2029079898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3e00b5c60 msgr2=0x7fe3e00b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3e00b5c60 0x7fe3e00b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 1 monclient: found mon.noname-b 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.801 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.796+0000 7fe3dffff640 1 -- 172.21.15.167:0/2029079898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e00b0640 con 0x7fe3e00b2f00 2024-04-29T20:21:58.802 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.798+0000 7fe3dffff640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 0x7fe3e00ae420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe3d00098c0 tx=0x7fe3d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:58.802 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.798+0000 7fe3dcff9640 10 client.?.objecter ms_handle_connect 0x7fe3e00b2f00 2024-04-29T20:21:58.802 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.798+0000 7fe3dcff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:58.803 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0043070 con 0x7fe3e00b2f00 2024-04-29T20:21:58.803 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.803 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: dump: 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: _finish_auth 0 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: _check_auth_tickets 2024-04-29T20:21:58.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d0033bb0 con 0x7fe3e00b2f00 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 5 monclient: authenticate success, global_id 4159 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 10 monclient: _renew_subs 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e00afa00 con 0x7fe3e00b2f00 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0038040 con 0x7fe3e00b2f00 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3dcff9640 10 monclient: dump: 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:58.805 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3de7fc640 4 set_mon_vals no callback set 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 10 monclient: _renew_subs 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e00aea30 con 0x7fe3e00b2f00 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.801+0000 7fe3e66f2640 1 librados: init done 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.802+0000 7fe3e66f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.802+0000 7fe3e66f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.806 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.802+0000 7fe3e66f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.807 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.802+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e00b8ae0 con 0x7fe3e00b2f00 2024-04-29T20:21:58.828 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-29T20:21:58.826+0000 7f06d7015040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-29T20:21:58.904 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe3d0046070 con 0x7fe3e00b2f00 2024-04-29T20:21:58.904 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe3d0052b60 con 0x7fe3e00b2f00 2024-04-29T20:21:58.904 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 10 client.4159.objecter ms_dispatch 0x7fe3e0000b90 osd_map(9..9 src has 1..9) 2024-04-29T20:21:58.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-04-29T20:21:58.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-04-29T20:21:58.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 20 client.4159.objecter dump_active .. 0 homeless 2024-04-29T20:21:58.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe3e00b8ae0 con 0x7fe3e00b2f00 2024-04-29T20:21:58.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:58.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.902+0000 7fe3dcff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:58.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.981+0000 7fe3e66f2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f4d72d02-c58f-4ff9-8b55-f6de73ba6460", "id": 7}] 2024-04-29T20:21:58.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.981+0000 7fe3e66f2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f4d72d02-c58f-4ff9-8b55-f6de73ba6460", "id": 7}] 2024-04-29T20:21:58.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.981+0000 7fe3e66f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:58.983 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:58.981+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f4d72d02-c58f-4ff9-8b55-f6de73ba6460", "id": 7} v 0) -- 0x7fe3e004fce0 con 0x7fe3e00b2f00 2024-04-29T20:21:59.170 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.169+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 2) ==== 45545+0+0 (secure 0 0 0) 0x7fe3d0052480 con 0x7fe3e00b2f00 2024-04-29T20:21:59.487 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3dcff9640 1 -- 172.21.15.167:0/2029079898 <== mon.1 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4d72d02-c58f-4ff9-8b55-f6de73ba6460", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe3d0030070 con 0x7fe3e00b2f00 2024-04-29T20:21:59.487 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f4d72d02-c58f-4ff9-8b55-f6de73ba6460", "id": 7}] 2024-04-29T20:21:59.487 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3dcff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:21:59.488 INFO:teuthology.orchestra.run.smithi167.stdout:7 2024-04-29T20:21:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 10 librados: watch_flush enter 2024-04-29T20:21:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 10 librados: watch_flush exit 2024-04-29T20:21:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 20 client.4159.objecter shutdown clearing up homeless session... 2024-04-29T20:21:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 10 client.4159.objecter successfully canceled tick 2024-04-29T20:21:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 10 monclient: shutdown 2024-04-29T20:21:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 msgr2=0x7fe3e00ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.486+0000 7fe3e66f2640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 0x7fe3e00ae420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe3d00098c0 tx=0x7fe3d00046c0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 shutdown_connections 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fe3e01d98a0 0x7fe3e01dc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3e00b5c60 0x7fe3e00b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 --2- 172.21.15.167:0/2029079898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3e00b2f00 0x7fe3e00ae420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 >> 172.21.15.167:0/2029079898 conn(0x7fe3e006c960 msgr2=0x7fe3e00aa460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:59.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 shutdown_connections 2024-04-29T20:21:59.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 -- 172.21.15.167:0/2029079898 wait complete. 2024-04-29T20:21:59.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-29T20:21:59.487+0000 7fe3e66f2640 1 librados: shutdown 2024-04-29T20:21:59.499 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.499 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-29T20:21:59.499 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-04-29T20:21:59.502 INFO:tasks.ceph.osd.0:Started 2024-04-29T20:21:59.502 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.502 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-29T20:21:59.502 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-04-29T20:21:59.504 INFO:tasks.ceph.osd.1:Started 2024-04-29T20:21:59.504 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.504 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-29T20:21:59.505 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-04-29T20:21:59.507 INFO:tasks.ceph.osd.2:Started 2024-04-29T20:21:59.507 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.508 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-29T20:21:59.508 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-04-29T20:21:59.512 INFO:tasks.ceph.osd.3:Started 2024-04-29T20:21:59.512 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.512 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-29T20:21:59.512 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-04-29T20:21:59.514 INFO:tasks.ceph.osd.4:Started 2024-04-29T20:21:59.514 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.515 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-29T20:21:59.515 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-04-29T20:21:59.517 INFO:tasks.ceph.osd.5:Started 2024-04-29T20:21:59.517 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.517 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-29T20:21:59.518 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-04-29T20:21:59.520 INFO:tasks.ceph.osd.6:Started 2024-04-29T20:21:59.520 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-04-29T20:21:59.520 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-29T20:21:59.520 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-04-29T20:21:59.523 INFO:tasks.ceph.osd.7:Started 2024-04-29T20:21:59.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-29T20:21:59.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.632+0000 7f8466915640 10 monclient: get_monmap_and_config 2024-04-29T20:21:59.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.632+0000 7f8466915640 10 monclient: build_initial_monmap 2024-04-29T20:21:59.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.633+0000 7f8466915640 10 monclient: monmap: 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:59.633889+0000 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:59.633889+0000 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:21:59.635 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:21:59.636 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:21:59.636 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:21:59.636 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:59.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 1 Processor -- start 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 1 -- start start 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 10 monclient: init 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84600b9570 addr v1:172.21.15.64:6789/0 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f84600b3240 0x7f8460140e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 10 monclient(hunting): picked mon.noname-c con 0x7f84600b3240 addr v2:172.21.15.167:3300/0 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8460146a90 addr v1:172.21.15.167:6789/0 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84600abd80 con 0x7f84600b9570 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.634+0000 7f8466915640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84600aac70 con 0x7f8460146a90 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f8466915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f8466915640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f84600b9950 con 0x7f84600b3240 2024-04-29T20:21:59.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f8466915640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f8466915640 10 monclient(hunting): authenticate will time out at 1468.794603s 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f845ffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f84600b9570 0x7f8460140720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47140/0 (socket says 172.21.15.64:47140) 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f845ffff640 1 -- 172.21.15.64:0/2183459945 learned_addr learned my addr 172.21.15.64:0/2183459945 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.635+0000 7f845f7fe640 1 --2- 172.21.15.64:0/2183459945 >> v2:172.21.15.167:3300/0 conn(0x7f84600b3240 0x7f8460140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.636+0000 7f845f7fe640 10 monclient(hunting): get_auth_request con 0x7f84600b3240 auth_method 0 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.636+0000 7f845f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.636+0000 7f845f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.636+0000 7f845f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.637+0000 7f845f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.637+0000 7f845f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:59.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.637+0000 7f845f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.638+0000 7f845effd640 1 -- 172.21.15.64:0/2183459945 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1841840861 0 0) 0x7f84600abd80 con 0x7f84600b9570 2024-04-29T20:21:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.638+0000 7f845effd640 10 monclient(hunting): _negotiate 2024-04-29T20:21:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.638+0000 7f845effd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.638+0000 7f845effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.638+0000 7f845effd640 10 monclient(hunting): my global_id is 4170 2024-04-29T20:21:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.638+0000 7f845effd640 1 -- 172.21.15.64:0/2183459945 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8448003520 con 0x7f84600b9570 2024-04-29T20:21:59.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 10 monclient(hunting): handle_auth_done global_id 4171 payload 274 2024-04-29T20:21:59.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 1 -- 172.21.15.64:0/2183459945 >> v1:172.21.15.167:6789/0 conn(0x7f8460146a90 legacy=0x7f8460146f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:59.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 1 -- 172.21.15.64:0/2183459945 >> v1:172.21.15.64:6789/0 conn(0x7f84600b9570 legacy=0x7f8460140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:59.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:21:59.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 1 monclient: found mon.noname-c 2024-04-29T20:21:59.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:59.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:21:59.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.640+0000 7f845f7fe640 1 -- 172.21.15.64:0/2183459945 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84600abff0 con 0x7f84600b3240 2024-04-29T20:21:59.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.644+0000 7f845f7fe640 1 --2- 172.21.15.64:0/2183459945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f8460140e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84540097a0 tx=0x7f845402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5044f301fc499d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 1 -- 172.21.15.64:0/2183459945 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454034070 con 0x7f84600b3240 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: dump: 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: _finish_auth 0 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: _check_auth_tickets 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 1 -- 172.21.15.64:0/2183459945 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8454030610 con 0x7f84600b3240 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 1 -- 172.21.15.64:0/2183459945 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84540309e0 con 0x7f84600b3240 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f845effd640 10 monclient: dump: 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:59.656 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f8466915640 5 monclient: authenticate success, global_id 4171 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f8466915640 10 monclient: get_monmap_and_config success 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.652+0000 7f8466915640 4 set_mon_vals no callback set 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 10 monclient: shutdown 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 -- 172.21.15.64:0/2183459945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 msgr2=0x7f8460140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 --2- 172.21.15.64:0/2183459945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f8460140e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84540097a0 tx=0x7f845402b960 comp rx=0 tx=0).stop 2024-04-29T20:21:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 -- 172.21.15.64:0/2183459945 shutdown_connections 2024-04-29T20:21:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 --2- 172.21.15.64:0/2183459945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f8460140e30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 -- 172.21.15.64:0/2183459945 >> 172.21.15.64:0/2183459945 conn(0x7f846006cba0 msgr2=0x7f846006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:21:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 -- 172.21.15.64:0/2183459945 shutdown_connections 2024-04-29T20:21:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.653+0000 7f8466915640 1 -- 172.21.15.64:0/2183459945 wait complete. 2024-04-29T20:21:59.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.655+0000 7f8466915640 10 monclient: build_initial_monmap 2024-04-29T20:21:59.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.655+0000 7f8466915640 10 monclient: monmap: 2024-04-29T20:21:59.661 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:21:59.661 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.655+0000 7f8466915640 1 librados: starting msgr at 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.655+0000 7f8466915640 1 librados: starting objecter 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.656+0000 7f8466915640 1 Processor -- start 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.656+0000 7f8466915640 1 -- start start 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.656+0000 7f8466915640 1 librados: setting wanted keys 2024-04-29T20:21:59.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.656+0000 7f8466915640 1 librados: calling monclient init 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.656+0000 7f8466915640 10 monclient: init 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.657+0000 7f8466915640 10 monclient: _reopen_session rank -1 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.657+0000 7f8466915640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.657+0000 7f8466915640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f84601391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.657+0000 7f8466915640 10 monclient(hunting): picked mon.noname-b con 0x7f84600b3240 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84600b9570 0x7f846013b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): picked mon.noname-a con 0x7f84600b9570 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8460146a90 0x7f84600aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): picked mon.noname-c con 0x7f8460146a90 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8460147790 con 0x7f84600b9570 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8460147610 con 0x7f84600b3240 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f845ffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f84601391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:21:59.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f846014edf0 con 0x7f8460146a90 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): _renew_subs 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.658+0000 7f8466915640 10 monclient(hunting): authenticate will time out at 1468.818479s 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f84601391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:41372/0 (socket says 172.21.15.64:41372) 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 1 -- 172.21.15.64:0/3619844196 learned_addr learned my addr 172.21.15.64:0/3619844196 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): get_auth_request con 0x7f84600b3240 auth_method 0 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:21:59.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.659+0000 7f845ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:21:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 10 monclient(hunting): handle_auth_done global_id 4174 payload 824 2024-04-29T20:21:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 1 -- 172.21.15.64:0/3619844196 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8460146a90 msgr2=0x7f84600aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8460146a90 0x7f84600aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 1 -- 172.21.15.64:0/3619844196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84600b9570 msgr2=0x7f846013b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:21:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84600b9570 0x7f846013b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:21:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 10 monclient: _finish_hunting 0 2024-04-29T20:21:59.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 1 monclient: found mon.noname-b 2024-04-29T20:21:59.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:21:59.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:21:59.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.664+0000 7f845ffff640 1 -- 172.21.15.64:0/3619844196 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846013cac0 con 0x7f84600b3240 2024-04-29T20:21:59.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.667+0000 7f845ffff640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f84601391d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f845000c430 tx=0x7f845000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:21:59.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.667+0000 7f845cff9640 10 client.?.objecter ms_handle_connect 0x7f84600b3240 2024-04-29T20:21:59.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.667+0000 7f845cff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450017070 con 0x7f84600b3240 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: dump: 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: _finish_auth 0 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: _check_auth_tickets 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8450005ce0 con 0x7f84600b3240 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84500072f0 con 0x7f84600b3240 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:21:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f845cff9640 10 monclient: dump: 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f8466915640 5 monclient: authenticate success, global_id 4174 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f8466915640 10 monclient: _renew_subs 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f8466915640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.673+0000 7f8466915640 1 -- 172.21.15.64:0/3619844196 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846013ba00 con 0x7f84600b3240 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.674+0000 7f8466915640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.674+0000 7f8466915640 10 monclient: _renew_subs 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.674+0000 7f8466915640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.674+0000 7f8466915640 1 -- 172.21.15.64:0/3619844196 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f846013a410 con 0x7f84600b3240 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.674+0000 7f8466915640 1 librados: init done 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.674+0000 7f845dffb640 4 set_mon_vals no callback set 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.675+0000 7f84467fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.675+0000 7f84467fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.675+0000 7f84467fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.675+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846004fce0 con 0x7f84600b3240 2024-04-29T20:21:59.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.694+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 2) ==== 45545+0+0 (secure 0 0 0) 0x7f845001a070 con 0x7f84600b3240 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8450010900 con 0x7f84600b3240 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 10 client.4174.objecter ms_dispatch 0x7f8460000b90 osd_map(10..10 src has 1..10) 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 3 client.4174.objecter handle_osd_map got epochs [10,10] > 0 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 3 client.4174.objecter handle_osd_map decoding full epoch 10 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 20 client.4174.objecter dump_active .. 0 homeless 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8450021020 con 0x7f84600b3240 2024-04-29T20:21:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:21:59.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.872+0000 7f845cff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:21:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.995+0000 7f84467fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:21:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.995+0000 7f84467fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:21:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.995+0000 7f84467fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:21:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:21:59.995+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8460137990 con 0x7f84600b3240 2024-04-29T20:22:00.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.295+0000 7f845cff9640 1 -- 172.21.15.64:0/3619844196 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8450019e10 con 0x7f84600b3240 2024-04-29T20:22:00.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.295+0000 7f845cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:00.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.295+0000 7f845cff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:22:00.298 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:21:59.194193+0000","last_up_change":"0.000000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:00.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 10 librados: watch_flush enter 2024-04-29T20:22:00.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 10 librados: watch_flush exit 2024-04-29T20:22:00.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 20 client.4174.objecter shutdown clearing up homeless session... 2024-04-29T20:22:00.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 10 client.4174.objecter successfully canceled tick 2024-04-29T20:22:00.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 10 monclient: shutdown 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 msgr2=0x7f84601391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f84601391d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f845000c430 tx=0x7f845000c8f0 comp rx=0 tx=0).stop 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 shutdown_connections 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8460146a90 0x7f84600aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f84600b9570 0x7f846013b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 --2- 172.21.15.64:0/3619844196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84600b3240 0x7f84601391d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 >> 172.21.15.64:0/3619844196 conn(0x7f846006cba0 msgr2=0x7f84600b60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 shutdown_connections 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 -- 172.21.15.64:0/3619844196 wait complete. 2024-04-29T20:22:00.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:00.297+0000 7f84467fc640 1 librados: shutdown 2024-04-29T20:22:00.315 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-29T20:22:00.315 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-29T20:22:00.914 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:22:00.913+0000 7f339e581640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-29T20:22:00.972 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:22:00.970+0000 7f339e581640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-29T20:22:01.121 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:01.266 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.265+0000 7f5770bd0640 10 monclient: get_monmap_and_config 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.265+0000 7f5770bd0640 10 monclient: build_initial_monmap 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.265+0000 7f5770bd0640 10 monclient: monmap: 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:01.265962+0000 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:01.265962+0000 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:01.267 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:01.269 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.266+0000 7f5770bd0640 1 Processor -- start 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.266+0000 7f5770bd0640 1 -- start start 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.266+0000 7f5770bd0640 10 monclient: init 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f576c072ea0 0x7f576c143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f576c072ea0 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f576c073280 addr v1:172.21.15.64:6789/0 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aa480 0x7f576c0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f576c0aa480 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f576c0ab2e0 con 0x7f576c073280 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:01.270 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f576c0aabe0 con 0x7f576c072ea0 2024-04-29T20:22:01.271 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:01.271 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f576c0b34d0 con 0x7f576c0aa480 2024-04-29T20:22:01.271 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.267+0000 7f5770bd0640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:01.271 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.268+0000 7f5770bd0640 10 monclient(hunting): authenticate will time out at 1470.427567s 2024-04-29T20:22:01.275 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.274+0000 7f576a59c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f576c072ea0 0x7f576c143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:01.275 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.274+0000 7f5769d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f576c073280 0x7f576c0b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47146/0 (socket says 172.21.15.64:47146) 2024-04-29T20:22:01.276 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.274+0000 7f5769d9b640 1 -- 172.21.15.64:0/3685611298 learned_addr learned my addr 172.21.15.64:0/3685611298 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:22:01.277 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.275+0000 7f576a59c640 10 monclient(hunting): get_auth_request con 0x7f576c072ea0 auth_method 0 2024-04-29T20:22:01.277 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.275+0000 7f576a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:01.277 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.275+0000 7f576a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:01.277 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.275+0000 7f576a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:01.279 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.277+0000 7f576a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:01.279 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.277+0000 7f576a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:01.279 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.277+0000 7f576a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:01.284 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 10 monclient(hunting): handle_auth_done global_id 4173 payload 274 2024-04-29T20:22:01.284 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 1 -- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aa480 msgr2=0x7f576c0b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:01.284 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 1 --2- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aa480 0x7f576c0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:01.284 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 1 -- 172.21.15.64:0/3685611298 >> v1:172.21.15.64:6789/0 conn(0x7f576c073280 legacy=0x7f576c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:01.284 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 10 monclient: _finish_hunting 0 2024-04-29T20:22:01.284 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 1 monclient: found mon.noname-a 2024-04-29T20:22:01.285 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:01.285 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:01.285 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.282+0000 7f576a59c640 1 -- 172.21.15.64:0/3685611298 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c0b5550 con 0x7f576c072ea0 2024-04-29T20:22:01.293 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.292+0000 7f576a59c640 1 --2- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c143490 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f575c0097a0 tx=0x7f575c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a11b1c6f71891b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 1 -- 172.21.15.64:0/3685611298 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c031780 con 0x7f576c072ea0 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: dump: 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:01.306 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: _finish_auth 0 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: _check_auth_tickets 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 1 -- 172.21.15.64:0/3685611298 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f575c037070 con 0x7f576c072ea0 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 1 -- 172.21.15.64:0/3685611298 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c031b40 con 0x7f576c072ea0 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f576959a640 10 monclient: dump: 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.307 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f5770bd0640 5 monclient: authenticate success, global_id 4173 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f5770bd0640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:01.308 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.304+0000 7f5770bd0640 4 set_mon_vals no callback set 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 10 monclient: shutdown 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 1 -- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 msgr2=0x7f576c143490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 1 --2- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c143490 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f575c0097a0 tx=0x7f575c02b820 comp rx=0 tx=0).stop 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 1 -- 172.21.15.64:0/3685611298 shutdown_connections 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 1 --2- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aa480 0x7f576c0b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 1 --2- 172.21.15.64:0/3685611298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c143490 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.305+0000 7f5770bd0640 1 -- 172.21.15.64:0/3685611298 >> 172.21.15.64:0/3685611298 conn(0x7f576c0b9020 msgr2=0x7f576c0b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.306+0000 7f5770bd0640 1 -- 172.21.15.64:0/3685611298 shutdown_connections 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.306+0000 7f5770bd0640 1 -- 172.21.15.64:0/3685611298 wait complete. 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.307+0000 7f5770bd0640 10 monclient: build_initial_monmap 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.307+0000 7f5770bd0640 10 monclient: monmap: 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:01.313 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.307+0000 7f5770bd0640 1 librados: starting msgr at 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.307+0000 7f5770bd0640 1 librados: starting objecter 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.308+0000 7f5770bd0640 1 Processor -- start 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.308+0000 7f5770bd0640 1 -- start start 2024-04-29T20:22:01.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.308+0000 7f5770bd0640 1 librados: setting wanted keys 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.308+0000 7f5770bd0640 1 librados: calling monclient init 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.308+0000 7f5770bd0640 10 monclient: init 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c13fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f576c072ea0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f576c0aa480 0x7f576c0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f576c0aa480 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aca20 0x7f576c0ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f576c0aca20 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f576c0aa8f0 con 0x7f576c072ea0 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.309+0000 7f5770bd0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f5770bd0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f576c0b2770 con 0x7f576c0aa480 2024-04-29T20:22:01.315 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f5770bd0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c13fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f5770bd0640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f576c14dcc0 con 0x7f576c0aca20 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f5770bd0640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f5770bd0640 10 monclient(hunting): authenticate will time out at 1470.469911s 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c13fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39876/0 (socket says 172.21.15.64:39876) 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 1 -- 172.21.15.64:0/1181329803 learned_addr learned my addr 172.21.15.64:0/1181329803 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 10 monclient(hunting): get_auth_request con 0x7f576c072ea0 auth_method 0 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f576a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.310+0000 7f5769d9b640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f576c0aa480 0x7f576c0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f5769d9b640 10 monclient(hunting): get_auth_request con 0x7f576c0aa480 auth_method 0 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f5769d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f5769d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:01.316 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f5769d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:01.317 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f576a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:01.317 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f576a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:01.317 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.311+0000 7f576a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:01.317 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.312+0000 7f5769d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:01.317 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.312+0000 7f5769d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:01.317 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.312+0000 7f5769d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 10 monclient(hunting): handle_auth_done global_id 4177 payload 824 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 1 -- 172.21.15.64:0/1181329803 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aca20 msgr2=0x7f576c0ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aca20 0x7f576c0ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 1 -- 172.21.15.64:0/1181329803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 msgr2=0x7f576c13fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c13fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 10 monclient: _finish_hunting 0 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 1 monclient: found mon.noname-b 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:01.319 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.317+0000 7f5769d9b640 1 -- 172.21.15.64:0/1181329803 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c0b05d0 con 0x7f576c0aa480 2024-04-29T20:22:01.322 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.320+0000 7f5769d9b640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f576c0aa480 0x7f576c0b0cd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f576000c440 tx=0x7f5760001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:01.322 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.320+0000 7f575affd640 10 client.?.objecter ms_handle_connect 0x7f576c0aa480 2024-04-29T20:22:01.322 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.320+0000 7f575affd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:01.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760017070 con 0x7f576c0aa480 2024-04-29T20:22:01.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:01.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:01.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: dump: 2024-04-29T20:22:01.329 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:01.329 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: _finish_auth 0 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5760005ce0 con 0x7f576c0aa480 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:01.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57600072d0 con 0x7f576c0aa480 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f575affd640 10 monclient: dump: 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.326+0000 7f5768d99640 4 set_mon_vals no callback set 2024-04-29T20:22:01.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 5 monclient: authenticate success, global_id 4177 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 10 monclient: _renew_subs 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576c0b15c0 con 0x7f576c0aa480 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 10 monclient: _renew_subs 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576c0ad3c0 con 0x7f576c0aa480 2024-04-29T20:22:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.327+0000 7f5770bd0640 1 librados: init done 2024-04-29T20:22:01.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.329+0000 7f5770bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:01.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.329+0000 7f5770bd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:01.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.329+0000 7f5770bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:01.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.329+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f576c0b13f0 con 0x7f576c0aa480 2024-04-29T20:22:01.369 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.367+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 3) ==== 90878+0+0 (secure 0 0 0) 0x7f576001a070 con 0x7f576c0aa480 2024-04-29T20:22:01.370 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.368+0000 7f575affd640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f574c06e0d0 0x7f574c070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:01.370 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.368+0000 7f576a59c640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f574c06e0d0 0x7f574c070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:01.370 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.368+0000 7f576a59c640 10 monclient: get_auth_request con 0x7f574c06e0d0 auth_method 0 2024-04-29T20:22:01.370 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.369+0000 7f575affd640 10 client.4177.objecter ms_handle_connect 0x7f574c06e0d0 2024-04-29T20:22:01.370 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.369+0000 7f576a59c640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f574c06e0d0 0x7f574c070590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f575c000c00 tx=0x7f575c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:01.825 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.823+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f5760057cb0 con 0x7f576c0aa480 2024-04-29T20:22:01.825 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.823+0000 7f575affd640 10 client.4177.objecter ms_dispatch 0x7f576c000b90 osd_map(10..10 src has 1..10) 2024-04-29T20:22:01.825 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.823+0000 7f575affd640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-04-29T20:22:01.825 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.823+0000 7f575affd640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-04-29T20:22:01.825 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.823+0000 7f575affd640 20 client.4177.objecter dump_active .. 0 homeless 2024-04-29T20:22:01.826 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.825+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5760059020 con 0x7f576c0aa480 2024-04-29T20:22:01.826 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.825+0000 7f575affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:01.827 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:01.825+0000 7f575affd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:02.214 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.212+0000 7f5770bd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:02.214 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.212+0000 7f5770bd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:02.214 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.212+0000 7f5770bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:02.214 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.212+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f576c0b1790 con 0x7f576c0aa480 2024-04-29T20:22:02.345 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.344+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f5760055690 con 0x7f576c0aa480 2024-04-29T20:22:02.507 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.505+0000 7f575affd640 1 -- 172.21.15.64:0/1181329803 <== mon.1 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f576003e4f0 con 0x7f576c0aa480 2024-04-29T20:22:02.507 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.505+0000 7f575affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:02.507 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.505+0000 7f575affd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:02.507 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:02.508 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:21:59.194193+0000","last_up_change":"0.000000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.508+0000 7f5770bd0640 10 librados: watch_flush enter 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.508+0000 7f5770bd0640 10 librados: watch_flush exit 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.508+0000 7f5770bd0640 20 client.4177.objecter shutdown clearing up homeless session... 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.508+0000 7f5770bd0640 10 client.4177.objecter successfully canceled tick 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f574c06e0d0 msgr2=0x7f574c070590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f574c06e0d0 0x7f574c070590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f575c000c00 tx=0x7f575c03d040 comp rx=0 tx=0).stop 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 10 monclient: shutdown 2024-04-29T20:22:02.510 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f576c0aa480 msgr2=0x7f576c0b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f576c0aa480 0x7f576c0b0cd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f576000c440 tx=0x7f5760001e70 comp rx=0 tx=0).stop 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 shutdown_connections 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f574c06e0d0 0x7f574c070590 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f576c0aca20 0x7f576c0ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f576c0aa480 0x7f576c0b0cd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 --2- 172.21.15.64:0/1181329803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f576c072ea0 0x7f576c13fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 >> 172.21.15.64:0/1181329803 conn(0x7f576c0b9020 msgr2=0x7f576c0b9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 shutdown_connections 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 -- 172.21.15.64:0/1181329803 wait complete. 2024-04-29T20:22:02.511 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:02.509+0000 7f5770bd0640 1 librados: shutdown 2024-04-29T20:22:02.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-29T20:22:06.975 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:07.395 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:06.980 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:07.394 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.037 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:07.449 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.038 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:07.458 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.045 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:07.459 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.107 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:07.514 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.279 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:07.701 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.338 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:07.760 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.451 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:07.857 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.512 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:07.917 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.581 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-29T20:22:07.572+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:07.613 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:08.040 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.618 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-29T20:22:07.610+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:07.621 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:08.024 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.677 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:08.100 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.678 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:22:07.667+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:07.684 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:08.088 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.797 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:08.206 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.857 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:08.266 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:07.940 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:22:07.927+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:08.058 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:22:08.051+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:08.066 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:08.487 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.093 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:08.507 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.219 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:08.632 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.248 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:08.241+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:08.252 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:08.245+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:08.445 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:08.438+0000 5dce500 -1 Falling back to public interface 2024-04-29T20:22:08.458 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:08.879 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.486 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:08.908 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.502 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:08.916 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.561 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:08.966 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.606 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.019 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.751 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.154 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.753 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.173 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.757 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:09.184 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.803 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:09.217 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.879 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.301 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.943 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.356 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.954 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.359 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:08.980 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:09.389 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.017 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.438 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.067 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:09.481 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.144 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.547 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.155 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:09.582 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.169 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.591 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.231 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.644 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.248 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.654 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.286 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.706 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.331 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:09.332 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:09.745 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.377 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:09.786 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.425 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.423+0000 7fa17ec31640 10 monclient: get_monmap_and_config 2024-04-29T20:22:09.425 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.423+0000 7fa17ec31640 10 monclient: build_initial_monmap 2024-04-29T20:22:09.425 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.423+0000 7fa17ec31640 10 monclient: monmap: 2024-04-29T20:22:09.425 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:09.425 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:09.424496+0000 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:09.424496+0000 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.424+0000 7fa17ec31640 1 Processor -- start 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.424+0000 7fa17ec31640 1 -- start start 2024-04-29T20:22:09.426 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.424+0000 7fa17ec31640 10 monclient: init 2024-04-29T20:22:09.428 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:09.428 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-29T20:22:09.428 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa178072ea0 0x7fa178143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:09.428 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): picked mon.noname-a con 0x7fa178072ea0 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:09.428 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa178073280 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:09.428 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa1780aa480 0x7fa1780b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): picked mon.noname-b con 0x7fa1780aa480 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1780ab2e0 con 0x7fa178073280 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fa1780aabe0 con 0x7fa178072ea0 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa1780b34d0 con 0x7fa1780aa480 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17ec31640 10 monclient(hunting): authenticate will time out at 1478.584711s 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17d1ce640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa1780aa480 0x7fa1780b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa17c9cd640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa178072ea0 0x7fa178143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa177fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fa178073280 0x7fa1780b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:36560/0 (socket says 172.21.15.64:36560) 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.425+0000 7fa177fff640 1 -- 172.21.15.64:0/4261520451 learned_addr learned my addr 172.21.15.64:0/4261520451 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): get_auth_request con 0x7fa1780aa480 auth_method 0 2024-04-29T20:22:09.429 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): get_auth_request con 0x7fa178072ea0 auth_method 0 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:09.430 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.426+0000 7fa17c9cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa1777fe640 1 -- 172.21.15.64:0/4261520451 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107273253 0 0) 0x7fa1780ab2e0 con 0x7fa178073280 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa1777fe640 10 monclient(hunting): _negotiate 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa1777fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa1777fe640 10 monclient(hunting): my global_id is 4204 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa1777fe640 1 -- 172.21.15.64:0/4261520451 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa168003520 con 0x7fa178073280 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_done global_id 4157 payload 274 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 1 -- 172.21.15.64:0/4261520451 >> v2:172.21.15.64:3300/0 conn(0x7fa178072ea0 msgr2=0x7fa178143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 1 --2- 172.21.15.64:0/4261520451 >> v2:172.21.15.64:3300/0 conn(0x7fa178072ea0 0x7fa178143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 1 -- 172.21.15.64:0/4261520451 >> v1:172.21.15.167:6789/0 conn(0x7fa178073280 legacy=0x7fa1780b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 10 monclient: _finish_hunting 0 2024-04-29T20:22:09.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 1 monclient: found mon.noname-b 2024-04-29T20:22:09.433 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:09.433 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:22:09.433 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.428+0000 7fa17d1ce640 1 -- 172.21.15.64:0/4261520451 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1780b5550 con 0x7fa1780aa480 2024-04-29T20:22:09.433 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.430+0000 7fa17d1ce640 1 --2- 172.21.15.64:0/4261520451 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa1780aa480 0x7fa1780b2020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa170009d90 tx=0x7fa17002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167bfad87133a4cc server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:09.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 1 -- 172.21.15.64:0/4261520451 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170039070 con 0x7fa1780aa480 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: dump: 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:09.435 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: _finish_auth 0 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: _check_auth_tickets 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 1 -- 172.21.15.64:0/4261520451 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa17002cbc0 con 0x7fa1780aa480 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa17ec31640 5 monclient: authenticate success, global_id 4157 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa17ec31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa17ec31640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa17ec31640 4 set_mon_vals no callback set 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 1 -- 172.21.15.64:0/4261520451 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170034440 con 0x7fa1780aa480 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.433+0000 7fa1777fe640 10 monclient: dump: 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:09.436 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:09.437 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:09.440 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 10 monclient: shutdown 2024-04-29T20:22:09.440 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:09.440 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 -- 172.21.15.64:0/4261520451 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa1780aa480 msgr2=0x7fa1780b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:09.440 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 --2- 172.21.15.64:0/4261520451 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa1780aa480 0x7fa1780b2020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa170009d90 tx=0x7fa17002be10 comp rx=0 tx=0).stop 2024-04-29T20:22:09.440 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 -- 172.21.15.64:0/4261520451 shutdown_connections 2024-04-29T20:22:09.440 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 --2- 172.21.15.64:0/4261520451 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa1780aa480 0x7fa1780b2020 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 --2- 172.21.15.64:0/4261520451 >> v2:172.21.15.64:3300/0 conn(0x7fa178072ea0 0x7fa178143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 -- 172.21.15.64:0/4261520451 >> 172.21.15.64:0/4261520451 conn(0x7fa1780b9020 msgr2=0x7fa1780b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 -- 172.21.15.64:0/4261520451 shutdown_connections 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.434+0000 7fa17ec31640 1 -- 172.21.15.64:0/4261520451 wait complete. 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 10 monclient: build_initial_monmap 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 10 monclient: monmap: 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:09.441 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:09.442 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:09.442 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 1 librados: starting msgr at 2024-04-29T20:22:09.442 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 1 librados: starting objecter 2024-04-29T20:22:09.442 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 1 Processor -- start 2024-04-29T20:22:09.442 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 1 -- start start 2024-04-29T20:22:09.443 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 1 librados: setting wanted keys 2024-04-29T20:22:09.443 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 1 librados: calling monclient init 2024-04-29T20:22:09.443 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.435+0000 7fa17ec31640 10 monclient: init 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa178072ea0 0x7fa17814cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): picked mon.noname-c con 0x7fa178072ea0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1780aa480 0x7fa17806e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): picked mon.noname-b con 0x7fa1780aa480 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa17806ecc0 0x7fa178138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): picked mon.noname-a con 0x7fa17806ecc0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa1780aa8f0 con 0x7fa17806ecc0 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1780b2770 con 0x7fa1780aa480 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa17814dcc0 con 0x7fa178072ea0 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17ec31640 10 monclient(hunting): authenticate will time out at 1478.595919s 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17c9cd640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa178072ea0 0x7fa17814cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa177fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1780aa480 0x7fa17806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.436+0000 7fa17d1ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa17806ecc0 0x7fa178138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa178072ea0 0x7fa17814cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:53120/0 (socket says 172.21.15.64:53120) 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 1 -- 172.21.15.64:0/2998224372 learned_addr learned my addr 172.21.15.64:0/2998224372 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): get_auth_request con 0x7fa178072ea0 auth_method 0 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): get_auth_request con 0x7fa1780aa480 auth_method 0 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:09.445 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17d1ce640 10 monclient(hunting): get_auth_request con 0x7fa17806ecc0 auth_method 0 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17d1ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17d1ce640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17d1ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa17c9cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.437+0000 7fa177fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.438+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.438+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:09.446 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.438+0000 7fa17d1ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2024-04-29T20:22:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 1 -- 172.21.15.64:0/2998224372 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa178072ea0 msgr2=0x7fa17814cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa178072ea0 0x7fa17814cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 1 -- 172.21.15.64:0/2998224372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa17806ecc0 msgr2=0x7fa178138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa17806ecc0 0x7fa178138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 10 monclient: _finish_hunting 0 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 1 monclient: found mon.noname-b 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.440+0000 7fa177fff640 1 -- 172.21.15.64:0/2998224372 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1781393f0 con 0x7fa1780aa480 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.442+0000 7fa177fff640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1780aa480 0x7fa17806e780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c400 tx=0x7fa16400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.442+0000 7fa1757fa640 10 client.?.objecter ms_handle_connect 0x7fa1780aa480 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.442+0000 7fa1757fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164017070 con 0x7fa1780aa480 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: dump: 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:09.449 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: _finish_auth 0 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: _check_auth_tickets 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164005ce0 con 0x7fa1780aa480 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1640072f0 con 0x7fa1780aa480 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 5 monclient: authenticate success, global_id 4207 2024-04-29T20:22:09.450 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 10 monclient: _renew_subs 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1781395c0 con 0x7fa1780aa480 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa176ffd640 4 set_mon_vals no callback set 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa1757fa640 10 monclient: dump: 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:09.451 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 10 monclient: _renew_subs 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1781dfe10 con 0x7fa1780aa480 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.446+0000 7fa17ec31640 1 librados: init done 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.447+0000 7fa17ec31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.447+0000 7fa17ec31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.447+0000 7fa17ec31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:09.452 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.447+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa178070420 con 0x7fa1780aa480 2024-04-29T20:22:09.452 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.873 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.466 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.465+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fa16401a070 con 0x7fa1780aa480 2024-04-29T20:22:09.466 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.465+0000 7fa1757fa640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa15c06e1f0 0x7fa15c0706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.466+0000 7fa17c9cd640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa15c06e1f0 0x7fa15c0706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.466+0000 7fa17c9cd640 10 monclient: get_auth_request con 0x7fa15c06e1f0 auth_method 0 2024-04-29T20:22:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.466+0000 7fa17c9cd640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa15c06e1f0 0x7fa15c0706b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009c00 tx=0x7fa16c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.466+0000 7fa1757fa640 10 client.4207.objecter ms_handle_connect 0x7fa15c06e1f0 2024-04-29T20:22:09.470 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.468+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa164057d10 con 0x7fa1780aa480 2024-04-29T20:22:09.470 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.468+0000 7fa1757fa640 10 client.4207.objecter ms_dispatch 0x7fa178000b90 osd_map(10..10 src has 1..10) 2024-04-29T20:22:09.470 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.468+0000 7fa1757fa640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-04-29T20:22:09.470 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.468+0000 7fa1757fa640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-04-29T20:22:09.470 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.468+0000 7fa1757fa640 20 client.4207.objecter dump_active .. 0 homeless 2024-04-29T20:22:09.484 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.887 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.499 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:09.926 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.516 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.929 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.526 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.932 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.562 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.982 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.602 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.016 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.716 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:10.125 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.716 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.138 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.717 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.716+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa164021020 con 0x7fa1780aa480 2024-04-29T20:22:09.717 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.716+0000 7fa1757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:09.717 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.716+0000 7fa1757fa640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:09.777 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:10.181 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.796 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:10.201 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.800 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:10.227 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.804 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.217 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.833 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:10.254 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.872 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.286 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.906+0000 7fa17ec31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.907+0000 7fa17ec31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.907+0000 7fa17ec31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.907+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa17806f390 con 0x7fa1780aa480 2024-04-29T20:22:09.925 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.924+0000 7fa1757fa640 1 -- 172.21.15.64:0/2998224372 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa164059080 con 0x7fa1780aa480 2024-04-29T20:22:09.926 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.924+0000 7fa1757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:09.926 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.924+0000 7fa1757fa640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:09.926 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:09.926 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:21:59.194193+0000","last_up_change":"0.000000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:09.928 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 10 librados: watch_flush enter 2024-04-29T20:22:09.928 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 10 librados: watch_flush exit 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 20 client.4207.objecter shutdown clearing up homeless session... 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 10 client.4207.objecter successfully canceled tick 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa15c06e1f0 msgr2=0x7fa15c0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa15c06e1f0 0x7fa15c0706b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009c00 tx=0x7fa16c009190 comp rx=0 tx=0).stop 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 10 monclient: shutdown 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1780aa480 msgr2=0x7fa17806e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1780aa480 0x7fa17806e780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c400 tx=0x7fa16400c8c0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 shutdown_connections 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa15c06e1f0 0x7fa15c0706b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa17806ecc0 0x7fa178138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1780aa480 0x7fa17806e780 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.926+0000 7fa17ec31640 1 --2- 172.21.15.64:0/2998224372 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa178072ea0 0x7fa17814cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:09.929 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.927+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 >> 172.21.15.64:0/2998224372 conn(0x7fa1780b9020 msgr2=0x7fa1781404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.927+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 shutdown_connections 2024-04-29T20:22:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.927+0000 7fa17ec31640 1 -- 172.21.15.64:0/2998224372 wait complete. 2024-04-29T20:22:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:09.927+0000 7fa17ec31640 1 librados: shutdown 2024-04-29T20:22:09.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-29T20:22:09.982 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.404 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.012 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:10.421 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.061 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:10.466 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.067 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:10.471 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.097 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.510 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.099 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:10.526 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.100 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:10.521 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.138 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.551 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.246 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.669 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.302 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:10.711 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.328 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:10.733 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.356 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:10.760 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.364 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:10.785 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.397 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.810 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.399 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:10.826 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.423 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.837 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.511 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.933 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.595 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.004 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.600 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.006 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.646 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:11.050 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.694 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.121 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.705 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:11.118 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.779 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:11.201 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.873 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.278 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.889 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:11.310 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.906 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.315 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.921 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:11.341 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.937 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:11.341 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.963 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:11.377 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.983 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.410 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:10.994 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:11.408 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.201 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.610 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.233 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:11.637 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.257 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:11.670 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.280 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.706 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.289 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:11.701 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.309 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:11.731 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.341 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:11.763 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.397 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.802 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.429 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.834 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.490 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.899 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.856 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:12.260 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.877 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:12.304 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.889 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:12.293 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:11.910 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:12.337 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:12.094 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:12.504 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:12.127 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:12.536 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:16.578 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:16.991 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:16.582 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:17.005 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:16.629 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:17.050 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:16.649 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:17.063 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:16.744 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:16.846 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.844+0000 7f32666e4640 10 monclient: get_monmap_and_config 2024-04-29T20:22:16.846 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.844+0000 7f32666e4640 10 monclient: build_initial_monmap 2024-04-29T20:22:16.846 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.845+0000 7f32666e4640 10 monclient: monmap: 2024-04-29T20:22:16.846 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:16.845849+0000 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:16.845849+0000 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:16.847 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.846+0000 7f32666e4640 1 Processor -- start 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.846+0000 7f32666e4640 1 -- start start 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.846+0000 7f32666e4640 10 monclient: init 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3260072ea0 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f3260073280 0x7f3260143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:16.851 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): picked mon.noname-c con 0x7f3260073280 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f32600aa480 0x7f32600b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): picked mon.noname-a con 0x7f32600aa480 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32600ab2e0 con 0x7f3260072ea0 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f32600aabe0 con 0x7f32600aa480 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f32600b34d0 con 0x7f3260073280 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f32666e4640 10 monclient(hunting): authenticate will time out at 1486.006916s 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.847+0000 7f3264c81640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f32600aa480 0x7f32600b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.848+0000 7f3264c81640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f32600aa480 0x7f32600b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48258/0 (socket says 172.21.15.64:48258) 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.848+0000 7f3264c81640 1 -- 172.21.15.64:0/2300780369 learned_addr learned my addr 172.21.15.64:0/2300780369 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:16.852 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.848+0000 7f3264c81640 10 monclient(hunting): get_auth_request con 0x7f32600aa480 auth_method 0 2024-04-29T20:22:16.853 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.848+0000 7f3264c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:16.853 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.848+0000 7f3264c81640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:16.853 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.848+0000 7f3264c81640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:16.853 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.849+0000 7f3264c81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:16.853 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.849+0000 7f3264c81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:16.853 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.849+0000 7f3264c81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:16.854 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.851+0000 7f3264c81640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.851+0000 7f3264c81640 1 -- 172.21.15.64:0/2300780369 >> v2:172.21.15.167:3300/0 conn(0x7f3260073280 msgr2=0x7f3260143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.851+0000 7f3264c81640 1 --2- 172.21.15.64:0/2300780369 >> v2:172.21.15.167:3300/0 conn(0x7f3260073280 0x7f3260143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.851+0000 7f3264c81640 1 -- 172.21.15.64:0/2300780369 >> v1:172.21.15.167:6789/0 conn(0x7f3260072ea0 legacy=0x7f32600b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.851+0000 7f3264c81640 10 monclient: _finish_hunting 0 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.851+0000 7f3264c81640 1 monclient: found mon.noname-a 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.852+0000 7f3264c81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.852+0000 7f3264c81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.852+0000 7f3264c81640 1 -- 172.21.15.64:0/2300780369 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32600b5550 con 0x7f32600aa480 2024-04-29T20:22:16.855 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.854+0000 7f3264c81640 1 --2- 172.21.15.64:0/2300780369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 0x7f32600b2020 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f32600abaf0 tx=0x7f325802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7c121b9924b0b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:16.859 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 1 -- 172.21.15.64:0/2300780369 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325802c9f0 con 0x7f32600aa480 2024-04-29T20:22:16.859 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:16.859 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:16.859 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 10 monclient: dump: 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 10 monclient: _finish_auth 0 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:16.860 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f325effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:16.861 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f32666e4640 5 monclient: authenticate success, global_id 4209 2024-04-29T20:22:16.861 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.857+0000 7f32666e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:16.863 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f325effd640 1 -- 172.21.15.64:0/2300780369 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325802cb90 con 0x7f32600aa480 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f325effd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f325effd640 1 -- 172.21.15.64:0/2300780369 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32580343f0 con 0x7f32600aa480 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f325effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f325effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f325effd640 10 monclient: dump: 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.864 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f32666e4640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.860+0000 7f32666e4640 4 set_mon_vals no callback set 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 10 monclient: shutdown 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 -- 172.21.15.64:0/2300780369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 msgr2=0x7f32600b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 --2- 172.21.15.64:0/2300780369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 0x7f32600b2020 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f32600abaf0 tx=0x7f325802be50 comp rx=0 tx=0).stop 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 -- 172.21.15.64:0/2300780369 shutdown_connections 2024-04-29T20:22:16.865 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 --2- 172.21.15.64:0/2300780369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 0x7f32600b2020 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:16.866 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 --2- 172.21.15.64:0/2300780369 >> v2:172.21.15.167:3300/0 conn(0x7f3260073280 0x7f3260143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:16.866 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 -- 172.21.15.64:0/2300780369 >> 172.21.15.64:0/2300780369 conn(0x7f32600b9020 msgr2=0x7f32600b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:16.866 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.861+0000 7f32666e4640 1 -- 172.21.15.64:0/2300780369 shutdown_connections 2024-04-29T20:22:16.866 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.862+0000 7f32666e4640 1 -- 172.21.15.64:0/2300780369 wait complete. 2024-04-29T20:22:16.869 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.863+0000 7f32666e4640 10 monclient: build_initial_monmap 2024-04-29T20:22:16.869 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.863+0000 7f32666e4640 10 monclient: monmap: 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:16.870 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.863+0000 7f32666e4640 1 librados: starting msgr at 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.863+0000 7f32666e4640 1 librados: starting objecter 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.864+0000 7f32666e4640 1 Processor -- start 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.864+0000 7f32666e4640 1 -- start start 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.864+0000 7f32666e4640 1 librados: setting wanted keys 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.864+0000 7f32666e4640 1 librados: calling monclient init 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.864+0000 7f32666e4640 10 monclient: init 2024-04-29T20:22:16.871 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.865+0000 7f32666e4640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.865+0000 7f32666e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.865+0000 7f32666e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3260072ea0 0x7f326013f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): picked mon.noname-b con 0x7f3260072ea0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 0x7f32600b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): picked mon.noname-a con 0x7f32600aa480 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 0x7f32600aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): picked mon.noname-c con 0x7f32600adfb0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:16.872 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f32600aa8f0 con 0x7f32600aa480 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f32600b2770 con 0x7f3260072ea0 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f326014dcc0 con 0x7f32600adfb0 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f3264c81640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 0x7f32600aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.866+0000 7f32666e4640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f32666e4640 10 monclient(hunting): authenticate will time out at 1486.026596s 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f3264c81640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 0x7f32600aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:59218/0 (socket says 172.21.15.64:59218) 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f3264c81640 1 -- 172.21.15.64:0/199227951 learned_addr learned my addr 172.21.15.64:0/199227951 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f3264c81640 10 monclient(hunting): get_auth_request con 0x7f32600adfb0 auth_method 0 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f3264c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f3264c81640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:16.873 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.867+0000 7f3264c81640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:16.874 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.868+0000 7f3264c81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:16.874 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.868+0000 7f3264c81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:16.874 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.868+0000 7f3264c81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:16.874 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.868+0000 7f325ffff640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3260072ea0 0x7f326013f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.869+0000 7f325ffff640 10 monclient(hunting): get_auth_request con 0x7f3260072ea0 auth_method 0 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.869+0000 7f325ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.869+0000 7f325ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.869+0000 7f325ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.869+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.869+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.870+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 1 -- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3260072ea0 msgr2=0x7f326013f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3260072ea0 0x7f326013f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 1 -- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 msgr2=0x7f32600b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 0x7f32600b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:16.876 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 10 monclient: _finish_hunting 0 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 1 monclient: found mon.noname-c 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.871+0000 7f3264c81640 1 -- 172.21.15.64:0/199227951 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32600b0330 con 0x7f32600adfb0 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.873+0000 7f3264c81640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 0x7f32600aed30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3258009f40 tx=0x7f3258004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.873+0000 7f325cff9640 10 client.?.objecter ms_handle_connect 0x7f32600adfb0 2024-04-29T20:22:16.877 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.873+0000 7f325cff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258043020 con 0x7f32600adfb0 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: dump: 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:16.881 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: _finish_auth 0 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: _check_auth_tickets 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325803e450 con 0x7f32600adfb0 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258009510 con 0x7f32600adfb0 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325cff9640 10 monclient: dump: 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:16.882 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:16.883 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.883 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:16.883 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:16.883 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:16.883 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:16.883 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f325e7fc640 4 set_mon_vals no callback set 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 5 monclient: authenticate success, global_id 4163 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 10 monclient: _renew_subs 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:16.884 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32600b0610 con 0x7f32600adfb0 2024-04-29T20:22:16.885 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:16.885 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 10 monclient: _renew_subs 2024-04-29T20:22:16.885 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:16.885 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32600af270 con 0x7f32600adfb0 2024-04-29T20:22:16.885 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.877+0000 7f32666e4640 1 librados: init done 2024-04-29T20:22:16.885 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.878+0000 7f32666e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:16.886 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.878+0000 7f32666e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:16.886 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.878+0000 7f32666e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:16.886 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:16.878+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32600ae950 con 0x7f32600adfb0 2024-04-29T20:22:17.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.153+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f3258046020 con 0x7f32600adfb0 2024-04-29T20:22:17.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.154+0000 7f325cff9640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f324c06e220 0x7f324c0706e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:17.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.154+0000 7f325ffff640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f324c06e220 0x7f324c0706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:17.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.154+0000 7f325ffff640 10 monclient: get_auth_request con 0x7f324c06e220 auth_method 0 2024-04-29T20:22:17.156 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.155+0000 7f325cff9640 10 client.4163.objecter ms_handle_connect 0x7f324c06e220 2024-04-29T20:22:17.156 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.155+0000 7f325ffff640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f324c06e220 0x7f324c0706e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32500017f0 tx=0x7f3250009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f325807a620 con 0x7f32600adfb0 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 10 client.4163.objecter ms_dispatch 0x7f3260000b90 osd_map(10..10 src has 1..10) 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 3 client.4163.objecter handle_osd_map got epochs [10,10] > 0 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 3 client.4163.objecter handle_osd_map decoding full epoch 10 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 20 client.4163.objecter dump_active .. 0 homeless 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f325804d070 con 0x7f32600adfb0 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:17.169 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.167+0000 7f325cff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:17.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.354+0000 7f32666e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:17.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.354+0000 7f32666e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:17.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.354+0000 7f32666e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:17.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.354+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3260068e10 con 0x7f32600adfb0 2024-04-29T20:22:17.356 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:17.761 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:17.406 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.404+0000 7f325cff9640 1 -- 172.21.15.64:0/199227951 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f32600b0330 con 0x7f32600adfb0 2024-04-29T20:22:17.406 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.404+0000 7f325cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:17.406 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.404+0000 7f325cff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:17.406 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:17.407 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:21:59.194193+0000","last_up_change":"0.000000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:17.409 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 10 librados: watch_flush enter 2024-04-29T20:22:17.409 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 10 librados: watch_flush exit 2024-04-29T20:22:17.409 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 20 client.4163.objecter shutdown clearing up homeless session... 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 10 client.4163.objecter successfully canceled tick 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f324c06e220 msgr2=0x7f324c0706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f324c06e220 0x7f324c0706e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32500017f0 tx=0x7f3250009040 comp rx=0 tx=0).stop 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 10 monclient: shutdown 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 msgr2=0x7f32600aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.407+0000 7f32666e4640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 0x7f32600aed30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3258009f40 tx=0x7f3258004a40 comp rx=0 tx=0).stop 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.408+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 shutdown_connections 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.408+0000 7f32666e4640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f324c06e220 0x7f324c0706e0 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32500017f0 tx=0x7f3250009040 comp rx=0 tx=0).stop 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.408+0000 7f32666e4640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f32600adfb0 0x7f32600aed30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:17.410 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.408+0000 7f32666e4640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32600aa480 0x7f32600b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:17.411 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.408+0000 7f32666e4640 1 --2- 172.21.15.64:0/199227951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3260072ea0 0x7f326013f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:17.411 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.408+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 >> 172.21.15.64:0/199227951 conn(0x7f32600b9020 msgr2=0x7f326013c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:17.412 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.410+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 shutdown_connections 2024-04-29T20:22:17.412 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.410+0000 7f32666e4640 1 -- 172.21.15.64:0/199227951 wait complete. 2024-04-29T20:22:17.412 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:17.410+0000 7f32666e4640 1 librados: shutdown 2024-04-29T20:22:17.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-29T20:22:18.034 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:18.443 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:18.060 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:18.486 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:18.315 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:18.718 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:21.884 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:22.296 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:21.888 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:22.301 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:22.325 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:22.747 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:22.329 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:22.751 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:22.395 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:22.809 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:22.399 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:22.814 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:22.417 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:22.837 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:22.421 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:22.842 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:23.039 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:23.444 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:23.046 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:23.451 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:23.752 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:24.161 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:23.756 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:24.165 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:24.043 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:24.469 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:24.047 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:24.474 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:24.108 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:24.510 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:24.111 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:24.515 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:24.231 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:24.325 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.322+0000 7f17a1950640 10 monclient: get_monmap_and_config 2024-04-29T20:22:24.325 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.322+0000 7f17a1950640 10 monclient: build_initial_monmap 2024-04-29T20:22:24.325 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.322+0000 7f17a1950640 10 monclient: monmap: 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:24.324435+0000 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:24.324435+0000 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.323+0000 7f17a1950640 1 Processor -- start 2024-04-29T20:22:24.326 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.323+0000 7f17a1950640 1 -- start start 2024-04-29T20:22:24.327 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.323+0000 7f17a1950640 10 monclient: init 2024-04-29T20:22:24.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:24.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-29T20:22:24.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c072ea0 0x7f179c143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:24.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): picked mon.noname-b con 0x7f179c072ea0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:24.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f179c073280 0x7f179c0aa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:24.329 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): picked mon.noname-c con 0x7f179c073280 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f179c0b2dc0 addr v1:172.21.15.64:6789/0 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f179c0b2730 con 0x7f179c0b2dc0 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f179c0aabe0 con 0x7f179c073280 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f179c0ab2e0 con 0x7f179c072ea0 2024-04-29T20:22:24.330 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.324+0000 7f17a1950640 10 monclient(hunting): authenticate will time out at 1493.485170s 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179affd640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c072ea0 0x7f179c143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f179c0b2dc0 0x7f179c0b2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:46128/0 (socket says 172.21.15.64:46128) 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179b7fe640 1 -- 172.21.15.64:0/3125125815 learned_addr learned my addr 172.21.15.64:0/3125125815 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179a7fc640 1 --2- 172.21.15.64:0/3125125815 >> v2:172.21.15.167:3300/0 conn(0x7f179c073280 0x7f179c0aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179affd640 10 monclient(hunting): get_auth_request con 0x7f179c072ea0 auth_method 0 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:24.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179a7fc640 10 monclient(hunting): get_auth_request con 0x7f179c073280 auth_method 0 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.325+0000 7f179a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.326+0000 7f179affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.326+0000 7f179affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.326+0000 7f179affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.326+0000 7f179a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.326+0000 7f179a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:24.332 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.326+0000 7f179a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 1 -- 172.21.15.64:0/3125125815 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c072ea0 msgr2=0x7f179c143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 1 --2- 172.21.15.64:0/3125125815 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c072ea0 0x7f179c143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 1 -- 172.21.15.64:0/3125125815 >> v1:172.21.15.64:6789/0 conn(0x7f179c0b2dc0 legacy=0x7f179c0b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 10 monclient: _finish_hunting 0 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 1 monclient: found mon.noname-c 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:22:24.333 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.328+0000 7f179a7fc640 1 -- 172.21.15.64:0/3125125815 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c0b5550 con 0x7f179c073280 2024-04-29T20:22:24.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.331+0000 7f179a7fc640 1 --2- 172.21.15.64:0/3125125815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c073280 0x7f179c0aa480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f17900097a0 tx=0x7f179002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b5d1f058fcc2c67 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 1 -- 172.21.15.64:0/3125125815 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790031780 con 0x7f179c073280 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: dump: 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:24.339 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: _finish_auth 0 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 1 -- 172.21.15.64:0/3125125815 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1790037070 con 0x7f179c073280 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f17a1950640 5 monclient: authenticate success, global_id 4213 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f17a1950640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f17a1950640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f17a1950640 4 set_mon_vals no callback set 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 1 -- 172.21.15.64:0/3125125815 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790031b40 con 0x7f179c073280 2024-04-29T20:22:24.340 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.335+0000 7f1799ffb640 10 monclient: dump: 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 10 monclient: shutdown 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 -- 172.21.15.64:0/3125125815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c073280 msgr2=0x7f179c0aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:24.341 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 --2- 172.21.15.64:0/3125125815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c073280 0x7f179c0aa480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f17900097a0 tx=0x7f179002b820 comp rx=0 tx=0).stop 2024-04-29T20:22:24.342 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 -- 172.21.15.64:0/3125125815 shutdown_connections 2024-04-29T20:22:24.342 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 --2- 172.21.15.64:0/3125125815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c073280 0x7f179c0aa480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:24.342 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 --2- 172.21.15.64:0/3125125815 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c072ea0 0x7f179c143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:24.342 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 -- 172.21.15.64:0/3125125815 >> 172.21.15.64:0/3125125815 conn(0x7f179c0b9020 msgr2=0x7f179c0b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:24.342 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 -- 172.21.15.64:0/3125125815 shutdown_connections 2024-04-29T20:22:24.342 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.336+0000 7f17a1950640 1 -- 172.21.15.64:0/3125125815 wait complete. 2024-04-29T20:22:24.347 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.337+0000 7f17a1950640 10 monclient: build_initial_monmap 2024-04-29T20:22:24.347 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.337+0000 7f17a1950640 10 monclient: monmap: 2024-04-29T20:22:24.347 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:24.347 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:24.347 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:24.348 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:24.348 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:24.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:24.348 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:24.348 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 1 librados: starting msgr at 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 1 librados: starting objecter 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 1 Processor -- start 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 1 -- start start 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 1 librados: setting wanted keys 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 1 librados: calling monclient init 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.338+0000 7f17a1950640 10 monclient: init 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.339+0000 7f17a1950640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.339+0000 7f17a1950640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f179c072ea0 0x7f179c13f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): picked mon.noname-a con 0x7f179c072ea0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:24.349 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 0x7f179c0ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): picked mon.noname-c con 0x7f179c0b2dc0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c0b0b60 0x7f179c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): picked mon.noname-b con 0x7f179c0b0b60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f179c14dcc0 con 0x7f179c072ea0 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f179c0b3460 con 0x7f179c0b0b60 2024-04-29T20:22:24.350 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f179c14de40 con 0x7f179c0b2dc0 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.340+0000 7f17a1950640 10 monclient(hunting): authenticate will time out at 1493.501215s 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.341+0000 7f179a7fc640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 0x7f179c0ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.342+0000 7f179a7fc640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 0x7f179c0ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:41086/0 (socket says 172.21.15.64:41086) 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.342+0000 7f179a7fc640 1 -- 172.21.15.64:0/544184491 learned_addr learned my addr 172.21.15.64:0/544184491 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.342+0000 7f179a7fc640 10 monclient(hunting): get_auth_request con 0x7f179c0b2dc0 auth_method 0 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.342+0000 7f179a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.342+0000 7f179a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.342+0000 7f179a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:24.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.344+0000 7f179b7fe640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c0b0b60 0x7f179c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.346+0000 7f179b7fe640 10 monclient(hunting): get_auth_request con 0x7f179c0b0b60 auth_method 0 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.346+0000 7f179b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.346+0000 7f179b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.346+0000 7f179b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.347+0000 7f179a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.347+0000 7f179a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:24.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.347+0000 7f179a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:24.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 10 monclient(hunting): handle_auth_done global_id 4169 payload 824 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 1 -- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c0b0b60 msgr2=0x7f179c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c0b0b60 0x7f179c0aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 1 -- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f179c072ea0 msgr2=0x7f179c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f179c072ea0 0x7f179c13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 10 monclient: _finish_hunting 0 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 1 monclient: found mon.noname-c 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:22:24.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.351+0000 7f179a7fc640 1 -- 172.21.15.64:0/544184491 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c0b01f0 con 0x7f179c0b2dc0 2024-04-29T20:22:24.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.355+0000 7f179a7fc640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 0x7f179c0ae0e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f179002be30 tx=0x7f17900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:24.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.355+0000 7f1783fff640 10 client.?.objecter ms_handle_connect 0x7f179c0b2dc0 2024-04-29T20:22:24.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.355+0000 7f1783fff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:24.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790032820 con 0x7f179c0b2dc0 2024-04-29T20:22:24.432 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 10 monclient: dump: 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:24.433 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 10 monclient: _finish_auth 0 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 10 monclient: _check_auth_tickets 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f1783fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 5 monclient: authenticate success, global_id 4169 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 10 monclient: _renew_subs 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c0b0460 con 0x7f179c0b2dc0 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 10 monclient: _renew_subs 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:24.434 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c0af130 con 0x7f179c0b2dc0 2024-04-29T20:22:24.435 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.429+0000 7f17a1950640 1 librados: init done 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.431+0000 7f17a1950640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.431+0000 7f17a1950640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.431+0000 7f17a1950640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.431+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f179c0b81a0 con 0x7f179c0b2dc0 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.432+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1790037070 con 0x7f179c0b2dc0 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.432+0000 7f1783fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:24.436 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.432+0000 7f17997fa640 4 set_mon_vals no callback set 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.434+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790053060 con 0x7f179c0b2dc0 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.434+0000 7f1783fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.434+0000 7f1783fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.434+0000 7f1783fff640 10 monclient: dump: 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:24.437 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:24.438 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:24.438 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:24.438 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:24.438 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:24.767 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.764+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f1790032bc0 con 0x7f179c0b2dc0 2024-04-29T20:22:24.767 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.765+0000 7f1783fff640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f178806e1f0 0x7f17880706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:24.768 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.765+0000 7f179affd640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f178806e1f0 0x7f17880706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:24.768 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.765+0000 7f179affd640 10 monclient: get_auth_request con 0x7f178806e1f0 auth_method 0 2024-04-29T20:22:24.768 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.765+0000 7f179affd640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f178806e1f0 0x7f17880706b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f178c006a90 tx=0x7f178c006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:24.768 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.765+0000 7f1783fff640 10 client.4169.objecter ms_handle_connect 0x7f178806e1f0 2024-04-29T20:22:24.801 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f17900ac490 con 0x7f179c0b2dc0 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 10 client.4169.objecter ms_dispatch 0x7f179c000b90 osd_map(10..10 src has 1..10) 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 20 client.4169.objecter dump_active .. 0 homeless 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1790040070 con 0x7f179c0b2dc0 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:24.802 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:24.799+0000 7f1783fff640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:25.030 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.027+0000 7f17a1950640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:25.030 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.027+0000 7f17a1950640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:25.030 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.027+0000 7f17a1950640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:25.030 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.027+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f179c0b8480 con 0x7f179c0b2dc0 2024-04-29T20:22:25.059 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.057+0000 7f1783fff640 1 -- 172.21.15.64:0/544184491 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f179002f050 con 0x7f179c0b2dc0 2024-04-29T20:22:25.060 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.057+0000 7f1783fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:25.060 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.057+0000 7f1783fff640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:25.060 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:25.060 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:21:59.194193+0000","last_up_change":"0.000000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 10 librados: watch_flush enter 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 10 librados: watch_flush exit 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 20 client.4169.objecter shutdown clearing up homeless session... 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 10 client.4169.objecter successfully canceled tick 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f178806e1f0 msgr2=0x7f17880706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f178806e1f0 0x7f17880706b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f178c006a90 tx=0x7f178c006a20 comp rx=0 tx=0).stop 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 10 monclient: shutdown 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 msgr2=0x7f179c0ae0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 0x7f179c0ae0e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f179002be30 tx=0x7f17900046c0 comp rx=0 tx=0).stop 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 shutdown_connections 2024-04-29T20:22:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f178806e1f0 0x7f17880706b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f179c0b2dc0 0x7f179c0ae0e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f179c0b0b60 0x7f179c0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 --2- 172.21.15.64:0/544184491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f179c072ea0 0x7f179c13f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 >> 172.21.15.64:0/544184491 conn(0x7f179c0b9020 msgr2=0x7f179c0b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 shutdown_connections 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 -- 172.21.15.64:0/544184491 wait complete. 2024-04-29T20:22:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:25.060+0000 7f17a1950640 1 librados: shutdown 2024-04-29T20:22:25.076 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-29T20:22:26.128 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:26.541 32632== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:26.417 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:26.839 32629== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:26.690 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:27.105 32631== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:26.748 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:27.169 32630== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:27.143 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:27.549 32895== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:28.332 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:28.735 32894== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:28.351 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:28.760 32893== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:28.515 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:28.942 32892== Warning: unimplemented fcntl command: 1038 2024-04-29T20:22:31.883 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:31.975 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.972+0000 7ff620811640 10 monclient: get_monmap_and_config 2024-04-29T20:22:31.975 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.972+0000 7ff620811640 10 monclient: build_initial_monmap 2024-04-29T20:22:31.976 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.972+0000 7ff620811640 10 monclient: monmap: 2024-04-29T20:22:31.976 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:31.976 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:31.976 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:31.974435+0000 2024-04-29T20:22:31.976 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:31.974435+0000 2024-04-29T20:22:31.976 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:31.977 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:31.978 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.973+0000 7ff620811640 1 Processor -- start 2024-04-29T20:22:31.978 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.973+0000 7ff620811640 1 -- start start 2024-04-29T20:22:31.978 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.973+0000 7ff620811640 10 monclient: init 2024-04-29T20:22:31.984 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.973+0000 7ff620811640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:31.984 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.973+0000 7ff620811640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-29T20:22:31.984 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 0x7ff61c13cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:31.984 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): picked mon.noname-b con 0x7ff61c0b5fc0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:31.984 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff61c0b5550 0x7ff61c13d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:31.984 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): picked mon.noname-c con 0x7ff61c0b5550 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff61c146b00 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff61c0ac110 con 0x7ff61c146b00 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff61c0b63a0 con 0x7ff61c0b5550 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff61c0aaf40 con 0x7ff61c0b5fc0 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:31.985 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff620811640 10 monclient(hunting): authenticate will time out at 1501.134827s 2024-04-29T20:22:31.986 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.974+0000 7ff61a59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 0x7ff61c13cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:31.986 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff619d9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff61c0b5550 0x7ff61c13d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:31.986 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 0x7ff61c13cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:41098/0 (socket says 172.21.15.64:41098) 2024-04-29T20:22:31.986 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 1 -- 172.21.15.64:0/3452417338 learned_addr learned my addr 172.21.15.64:0/3452417338 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:31.986 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): get_auth_request con 0x7ff61c0b5fc0 auth_method 0 2024-04-29T20:22:31.986 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff619d9b640 10 monclient(hunting): get_auth_request con 0x7ff61c0b5550 auth_method 0 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff619d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff619d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.975+0000 7ff61a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.976+0000 7ff619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.976+0000 7ff619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:31.987 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.976+0000 7ff619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:31.988 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2024-04-29T20:22:31.988 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 1 -- 172.21.15.64:0/3452417338 >> v2:172.21.15.167:3300/0 conn(0x7ff61c0b5550 msgr2=0x7ff61c13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:31.988 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 1 --2- 172.21.15.64:0/3452417338 >> v2:172.21.15.167:3300/0 conn(0x7ff61c0b5550 0x7ff61c13d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:31.988 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 1 -- 172.21.15.64:0/3452417338 >> v1:172.21.15.167:6789/0 conn(0x7ff61c146b00 legacy=0x7ff61c146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:31.988 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 10 monclient: _finish_hunting 0 2024-04-29T20:22:31.988 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 1 monclient: found mon.noname-b 2024-04-29T20:22:31.989 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:31.989 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:22:31.989 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.977+0000 7ff61a59c640 1 -- 172.21.15.64:0/3452417338 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c0ac380 con 0x7ff61c0b5fc0 2024-04-29T20:22:31.989 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.979+0000 7ff61a59c640 1 --2- 172.21.15.64:0/3452417338 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 0x7ff61c13cd30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0098a0 tx=0x7ff60c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a53059a67ddcd443 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:31.996 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 1 -- 172.21.15.64:0/3452417338 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c0317c0 con 0x7ff61c0b5fc0 2024-04-29T20:22:31.996 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:31.996 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: dump: 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: _finish_auth 0 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: _check_auth_tickets 2024-04-29T20:22:31.997 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 1 -- 172.21.15.64:0/3452417338 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff60c037070 con 0x7ff61c0b5fc0 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 5 monclient: authenticate success, global_id 4172 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 1 -- 172.21.15.64:0/3452417338 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c031b80 con 0x7ff61c0b5fc0 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 4 set_mon_vals no callback set 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff61959a640 10 monclient: dump: 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:31.998 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 10 monclient: shutdown 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 1 -- 172.21.15.64:0/3452417338 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 msgr2=0x7ff61c13cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.983+0000 7ff620811640 1 --2- 172.21.15.64:0/3452417338 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 0x7ff61c13cd30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0098a0 tx=0x7ff60c02b860 comp rx=0 tx=0).stop 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.984+0000 7ff620811640 1 -- 172.21.15.64:0/3452417338 shutdown_connections 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.984+0000 7ff620811640 1 --2- 172.21.15.64:0/3452417338 >> v2:172.21.15.167:3300/0 conn(0x7ff61c0b5550 0x7ff61c13d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.984+0000 7ff620811640 1 --2- 172.21.15.64:0/3452417338 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5fc0 0x7ff61c13cd30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:31.999 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.984+0000 7ff620811640 1 -- 172.21.15.64:0/3452417338 >> 172.21.15.64:0/3452417338 conn(0x7ff61c062570 msgr2=0x7ff61c062980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.984+0000 7ff620811640 1 -- 172.21.15.64:0/3452417338 shutdown_connections 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.984+0000 7ff620811640 1 -- 172.21.15.64:0/3452417338 wait complete. 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.985+0000 7ff620811640 10 monclient: build_initial_monmap 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.985+0000 7ff620811640 10 monclient: monmap: 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:32.000 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.985+0000 7ff620811640 1 librados: starting msgr at 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.985+0000 7ff620811640 1 librados: starting objecter 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 Processor -- start 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 -- start start 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 librados: setting wanted keys 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 librados: calling monclient init 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient: init 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:22:32.001 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 0x7ff61c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): picked mon.noname-c con 0x7ff61c0b5550 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff61c146b00 0x7ff61c0b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): picked mon.noname-a con 0x7ff61c146b00 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff61c148050 0x7ff61c1dbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): picked mon.noname-b con 0x7ff61c148050 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c0aacd0 con 0x7ff61c146b00 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c0aab50 con 0x7ff61c148050 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff61c14dbb0 con 0x7ff61c0b5550 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:32.002 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.986+0000 7ff620811640 10 monclient(hunting): authenticate will time out at 1501.147345s 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 0x7ff61c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff619d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff61c146b00 0x7ff61c0b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 0x7ff61c0b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:41104/0 (socket says 172.21.15.64:41104) 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 1 -- 172.21.15.64:0/3216022125 learned_addr learned my addr 172.21.15.64:0/3216022125 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 10 monclient(hunting): get_auth_request con 0x7ff61c0b5550 auth_method 0 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61ad9d640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff61c148050 0x7ff61c1dbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.987+0000 7ff61ad9d640 10 monclient(hunting): get_auth_request con 0x7ff61c148050 auth_method 0 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:32.003 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): get_auth_request con 0x7ff61c146b00 auth_method 0 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff61ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.988+0000 7ff619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 10 monclient(hunting): handle_auth_done global_id 4175 payload 824 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 1 -- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff61c148050 msgr2=0x7ff61c1dbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:32.004 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff61c148050 0x7ff61c1dbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 1 -- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff61c146b00 msgr2=0x7ff61c0b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff61c146b00 0x7ff61c0b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 10 monclient: _finish_hunting 0 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 1 monclient: found mon.noname-c 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:22:32.005 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.992+0000 7ff61a59c640 1 -- 172.21.15.64:0/3216022125 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c1492c0 con 0x7ff61c0b5550 2024-04-29T20:22:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.995+0000 7ff61a59c640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 0x7ff61c0b99b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff60c02bdf0 tx=0x7ff60c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.995+0000 7ff60affd640 10 client.?.objecter ms_handle_connect 0x7ff61c0b5550 2024-04-29T20:22:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.995+0000 7ff60affd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c046650 con 0x7ff61c0b5550 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 10 monclient: dump: 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:32.008 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 10 monclient: _finish_auth 0 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff60affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 5 monclient: authenticate success, global_id 4175 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 10 monclient: _renew_subs 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61c149530 con 0x7ff61c0b5550 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 10 monclient: _renew_subs 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61c148510 con 0x7ff61c0b5550 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.998+0000 7ff620811640 1 librados: init done 2024-04-29T20:22:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff60c0467f0 con 0x7ff61c0b5550 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff60affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff620811640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c046ba0 con 0x7ff61c0b5550 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff620811640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff620811640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff618d99640 4 set_mon_vals no callback set 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff61c06b290 con 0x7ff61c0b5550 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff60affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff60affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:32.010 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:31.999+0000 7ff60affd640 10 monclient: dump: 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:32.011 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:32.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.310+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7ff60c032450 con 0x7ff61c0b5550 2024-04-29T20:22:32.313 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.311+0000 7ff60affd640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff5fc06e1f0 0x7ff5fc0706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:32.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.311+0000 7ff619d9b640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff5fc06e1f0 0x7ff5fc0706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:32.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.311+0000 7ff619d9b640 10 monclient: get_auth_request con 0x7ff5fc06e1f0 auth_method 0 2024-04-29T20:22:32.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.312+0000 7ff60affd640 10 client.4175.objecter ms_handle_connect 0x7ff5fc06e1f0 2024-04-29T20:22:32.314 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.312+0000 7ff619d9b640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff5fc06e1f0 0x7ff5fc0706b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff61c14d570 tx=0x7ff610009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:32.327 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff60c0ac2b0 con 0x7ff61c0b5550 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 10 client.4175.objecter ms_dispatch 0x7ff61c000b90 osd_map(10..10 src has 1..10) 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 3 client.4175.objecter handle_osd_map decoding full epoch 10 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 20 client.4175.objecter dump_active .. 0 homeless 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff60c07bb30 con 0x7ff61c0b5550 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.325+0000 7ff60affd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:32.687 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.684+0000 7ff620811640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:32.687 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.684+0000 7ff620811640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:32.687 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.684+0000 7ff620811640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:32.687 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.684+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff61c1498c0 con 0x7ff61c0b5550 2024-04-29T20:22:32.704 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.701+0000 7ff60affd640 1 -- 172.21.15.64:0/3216022125 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff60c07fe30 con 0x7ff61c0b5550 2024-04-29T20:22:32.704 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.701+0000 7ff60affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:32.704 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.701+0000 7ff60affd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:32.704 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:32.705 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:21:59.194193+0000","last_up_change":"0.000000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:32.708 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 10 librados: watch_flush enter 2024-04-29T20:22:32.708 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 10 librados: watch_flush exit 2024-04-29T20:22:32.708 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 20 client.4175.objecter shutdown clearing up homeless session... 2024-04-29T20:22:32.708 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 10 client.4175.objecter successfully canceled tick 2024-04-29T20:22:32.708 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff5fc06e1f0 msgr2=0x7ff5fc0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:32.708 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff5fc06e1f0 0x7ff5fc0706b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff61c14d570 tx=0x7ff610009190 comp rx=0 tx=0).stop 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 10 monclient: shutdown 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 msgr2=0x7ff61c0b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 0x7ff61c0b99b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff60c02bdf0 tx=0x7ff60c0046c0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 shutdown_connections 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff5fc06e1f0 0x7ff5fc0706b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff61c148050 0x7ff61c1dbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff61c146b00 0x7ff61c0b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 --2- 172.21.15.64:0/3216022125 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff61c0b5550 0x7ff61c0b99b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 >> 172.21.15.64:0/3216022125 conn(0x7ff61c062570 msgr2=0x7ff61c0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.704+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 shutdown_connections 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.705+0000 7ff620811640 1 -- 172.21.15.64:0/3216022125 wait complete. 2024-04-29T20:22:32.709 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:32.705+0000 7ff620811640 1 librados: shutdown 2024-04-29T20:22:32.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-29T20:22:33.797 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:22:33.791+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-04-29T20:22:34.539 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:22:34.534+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-04-29T20:22:34.700 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-29T20:22:34.697+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-04-29T20:22:34.838 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-29T20:22:34.834+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-04-29T20:22:35.961 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:22:35.955+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-04-29T20:22:37.445 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:37.441+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-04-29T20:22:37.479 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:37.475+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-04-29T20:22:38.061 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:38.055+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-04-29T20:22:38.072 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-29T20:22:38.067+0000 bc99640 -1 osd.7 0 waiting for initial osdmap 2024-04-29T20:22:38.087 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:22:38.082+0000 b485640 -1 osd.4 0 waiting for initial osdmap 2024-04-29T20:22:38.136 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-29T20:22:38.131+0000 bc99640 -1 osd.5 0 waiting for initial osdmap 2024-04-29T20:22:38.538 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-29T20:22:38.522+0000 10260640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:38.591 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-29T20:22:38.569+0000 10260640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:38.597 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:22:38.581+0000 10260640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:39.193 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:22:39.189+0000 bc99640 -1 osd.2 0 waiting for initial osdmap 2024-04-29T20:22:39.527 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:39.601 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:22:39.585+0000 10260640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.623+0000 7f9684a5c640 10 monclient: get_monmap_and_config 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.623+0000 7f9684a5c640 10 monclient: build_initial_monmap 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.623+0000 7f9684a5c640 10 monclient: monmap: 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:39.624803+0000 2024-04-29T20:22:39.625 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:39.624803+0000 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:39.626 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:39.628 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.624+0000 7f9684a5c640 1 Processor -- start 2024-04-29T20:22:39.628 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.624+0000 7f9684a5c640 1 -- start start 2024-04-29T20:22:39.628 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.624+0000 7f9684a5c640 10 monclient: init 2024-04-29T20:22:39.628 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f9680072ea0 0x7f9680143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9680072ea0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9680073280 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f96800aa480 0x7f96800b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): picked mon.noname-c con 0x7f96800aa480 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96800ab2e0 con 0x7f9680073280 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f96800aabe0 con 0x7f96800aa480 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f96800aa910 con 0x7f9680072ea0 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f9684a5c640 10 monclient(hunting): authenticate will time out at 1508.784977s 2024-04-29T20:22:39.629 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.625+0000 7f967ed9d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f96800aa480 0x7f96800b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f9680072ea0 0x7f9680143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967ed9d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f96800aa480 0x7f96800b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:47782/0 (socket says 172.21.15.64:47782) 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967ed9d640 1 -- 172.21.15.64:0/2973351440 learned_addr learned my addr 172.21.15.64:0/2973351440 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967ed9d640 10 monclient(hunting): get_auth_request con 0x7f96800aa480 auth_method 0 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 10 monclient(hunting): get_auth_request con 0x7f9680072ea0 auth_method 0 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:39.630 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.626+0000 7f967e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:39.631 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.627+0000 7f967e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:39.631 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.627+0000 7f967ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:39.631 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.627+0000 7f967ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:39.631 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.627+0000 7f967ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 1 -- 172.21.15.64:0/2973351440 >> v2:172.21.15.167:3300/0 conn(0x7f96800aa480 msgr2=0x7f96800b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 1 --2- 172.21.15.64:0/2973351440 >> v2:172.21.15.167:3300/0 conn(0x7f96800aa480 0x7f96800b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 1 -- 172.21.15.64:0/2973351440 >> v1:172.21.15.167:6789/0 conn(0x7f9680073280 legacy=0x7f96800b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 10 monclient: _finish_hunting 0 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 1 monclient: found mon.noname-b 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.628+0000 7f967e59c640 1 -- 172.21.15.64:0/2973351440 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96800b5550 con 0x7f9680072ea0 2024-04-29T20:22:39.632 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.630+0000 7f967e59c640 1 --2- 172.21.15.64:0/2973351440 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f9680072ea0 0x7f9680143490 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96700097a0 tx=0x7f967002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb62b96daf89561 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 1 -- 172.21.15.64:0/2973351440 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96700315a0 con 0x7f9680072ea0 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: dump: 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.637 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: _finish_auth 0 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: _check_auth_tickets 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 1 -- 172.21.15.64:0/2973351440 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9670037070 con 0x7f9680072ea0 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 1 -- 172.21.15.64:0/2973351440 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9670031960 con 0x7f9680072ea0 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f9684a5c640 5 monclient: authenticate success, global_id 4202 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f9684a5c640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:39.638 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f9684a5c640 4 set_mon_vals no callback set 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.635+0000 7f967d59a640 10 monclient: dump: 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:39.639 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 10 monclient: shutdown 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 1 -- 172.21.15.64:0/2973351440 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f9680072ea0 msgr2=0x7f9680143490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2973351440 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f9680072ea0 0x7f9680143490 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96700097a0 tx=0x7f967002b840 comp rx=0 tx=0).stop 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 1 -- 172.21.15.64:0/2973351440 shutdown_connections 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2973351440 >> v2:172.21.15.167:3300/0 conn(0x7f96800aa480 0x7f96800b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2973351440 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f9680072ea0 0x7f9680143490 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.636+0000 7f9684a5c640 1 -- 172.21.15.64:0/2973351440 >> 172.21.15.64:0/2973351440 conn(0x7f96800b9020 msgr2=0x7f96800b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:39.642 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.637+0000 7f9684a5c640 1 -- 172.21.15.64:0/2973351440 shutdown_connections 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.637+0000 7f9684a5c640 1 -- 172.21.15.64:0/2973351440 wait complete. 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.637+0000 7f9684a5c640 10 monclient: build_initial_monmap 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.637+0000 7f9684a5c640 10 monclient: monmap: 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:39.643 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 librados: starting msgr at 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 librados: starting objecter 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 Processor -- start 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 -- start start 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 librados: setting wanted keys 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 librados: calling monclient init 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient: init 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9680072ea0 0x7f96800b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9680072ea0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96800aa480 0x7f968013f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): picked mon.noname-a con 0x7f96800aa480 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:39.644 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 0x7f96800aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): picked mon.noname-c con 0x7f96800ae050 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f96800b34d0 con 0x7f96800aa480 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f96800b2770 con 0x7f9680072ea0 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f968014dcc0 con 0x7f96800ae050 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.638+0000 7f9684a5c640 10 monclient(hunting): authenticate will time out at 1508.798398s 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 0x7f96800aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 0x7f96800aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:47190/0 (socket says 172.21.15.64:47190) 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 1 -- 172.21.15.64:0/2714876293 learned_addr learned my addr 172.21.15.64:0/2714876293 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 10 monclient(hunting): get_auth_request con 0x7f96800ae050 auth_method 0 2024-04-29T20:22:39.645 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:39.646 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:39.646 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.639+0000 7f967ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:39.646 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.640+0000 7f967ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:39.646 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.640+0000 7f967ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:39.646 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.640+0000 7f967ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967e59c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9680072ea0 0x7f96800b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 1 -- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9680072ea0 msgr2=0x7f96800b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9680072ea0 0x7f96800b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 1 -- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96800aa480 msgr2=0x7f968013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96800aa480 0x7f968013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 10 monclient: _finish_hunting 0 2024-04-29T20:22:39.647 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 1 monclient: found mon.noname-c 2024-04-29T20:22:39.648 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:39.648 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:22:39.648 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.643+0000 7f967ed9d640 1 -- 172.21.15.64:0/2714876293 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96800b0440 con 0x7f96800ae050 2024-04-29T20:22:39.648 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.646+0000 7f967ed9d640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 0x7f96800aecc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f967800ec10 tx=0x7f967800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:39.648 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.646+0000 7f966effd640 10 client.?.objecter ms_handle_connect 0x7f96800ae050 2024-04-29T20:22:39.648 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.646+0000 7f966effd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:39.651 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967800cb20 con 0x7f96800ae050 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: dump: 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:39.652 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: _finish_auth 0 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f967800ccc0 con 0x7f96800ae050 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 5 monclient: authenticate success, global_id 4205 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9678003bb0 con 0x7f96800ae050 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 10 monclient: _renew_subs 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96800b0720 con 0x7f96800ae050 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f967cd99640 4 set_mon_vals no callback set 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:39.653 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f966effd640 10 monclient: dump: 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 10 monclient: _renew_subs 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:39.654 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96800af3e0 con 0x7f96800ae050 2024-04-29T20:22:39.655 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.650+0000 7f9684a5c640 1 librados: init done 2024-04-29T20:22:39.655 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.651+0000 7f9684a5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:39.655 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.651+0000 7f9684a5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:39.655 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.651+0000 7f9684a5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:39.655 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.651+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96800ae660 con 0x7f96800ae050 2024-04-29T20:22:39.674 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.672+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f9678005d10 con 0x7f96800ae050 2024-04-29T20:22:39.674 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.673+0000 7f966effd640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f966006e1f0 0x7f96600706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:39.675 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.673+0000 7f967e59c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f966006e1f0 0x7f96600706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:39.675 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.673+0000 7f967e59c640 10 monclient: get_auth_request con 0x7f966006e1f0 auth_method 0 2024-04-29T20:22:39.675 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.674+0000 7f966effd640 10 client.4205.objecter ms_handle_connect 0x7f966006e1f0 2024-04-29T20:22:39.675 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.674+0000 7f967e59c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f966006e1f0 0x7f96600706b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9670036000 tx=0x7f9670008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:39.677 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.676+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 2120+0+0 (secure 0 0 0) 0x7f9678058db0 con 0x7f96800ae050 2024-04-29T20:22:39.677 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.676+0000 7f966effd640 10 client.4205.objecter ms_dispatch 0x7f9680000b90 osd_map(14..14 src has 1..14) 2024-04-29T20:22:39.677 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.676+0000 7f966effd640 3 client.4205.objecter handle_osd_map got epochs [14,14] > 0 2024-04-29T20:22:39.678 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.676+0000 7f966effd640 3 client.4205.objecter handle_osd_map decoding full epoch 14 2024-04-29T20:22:39.678 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.676+0000 7f966effd640 20 client.4205.objecter dump_active .. 0 homeless 2024-04-29T20:22:39.964 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.963+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9678010040 con 0x7f96800ae050 2024-04-29T20:22:39.965 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.963+0000 7f966effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:39.965 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:39.963+0000 7f966effd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:40.157 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.156+0000 7f9684a5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:40.157 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.156+0000 7f9684a5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:40.157 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.156+0000 7f9684a5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:22:40.157 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.156+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96800ae880 con 0x7f96800ae050 2024-04-29T20:22:40.395 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.394+0000 7f966effd640 1 -- 172.21.15.64:0/2714876293 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9013 (secure 0 0 0) 0x7f96800ae880 con 0x7f96800ae050 2024-04-29T20:22:40.395 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.394+0000 7f966effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:40.396 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.394+0000 7f966effd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:40.396 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:40.396 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":15,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:22:39.830788+0000","last_up_change":"2024-04-29T20:22:39.830788+0000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6801","nonce":2373308255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6803","nonce":2373308255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6807","nonce":2373308255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6805","nonce":2373308255}]},"public_addr":"172.21.15.64:6801/2373308255","cluster_addr":"172.21.15.64:6803/2373308255","heartbeat_back_addr":"172.21.15.64:6807/2373308255","heartbeat_front_addr":"172.21.15.64:6805/2373308255","state":["exists","up"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6827","nonce":3754201859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6829","nonce":3754201859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6833","nonce":3754201859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6831","nonce":3754201859}]},"public_addr":"172.21.15.167:6827/3754201859","cluster_addr":"172.21.15.167:6829/3754201859","heartbeat_back_addr":"172.21.15.167:6833/3754201859","heartbeat_front_addr":"172.21.15.167:6831/3754201859","state":["exists","up"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6803","nonce":3447508053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6805","nonce":3447508053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6809","nonce":3447508053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6807","nonce":3447508053}]},"public_addr":"172.21.15.167:6803/3447508053","cluster_addr":"172.21.15.167:6805/3447508053","heartbeat_back_addr":"172.21.15.167:6809/3447508053","heartbeat_front_addr":"172.21.15.167:6807/3447508053","state":["exists","up"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6811","nonce":2258625350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6813","nonce":2258625350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6817","nonce":2258625350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6815","nonce":2258625350}]},"public_addr":"172.21.15.167:6811/2258625350","cluster_addr":"172.21.15.167:6813/2258625350","heartbeat_back_addr":"172.21.15.167:6817/2258625350","heartbeat_front_addr":"172.21.15.167:6815/2258625350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:40.398 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 10 librados: watch_flush enter 2024-04-29T20:22:40.398 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 10 librados: watch_flush exit 2024-04-29T20:22:40.398 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 20 client.4205.objecter shutdown clearing up homeless session... 2024-04-29T20:22:40.398 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 10 client.4205.objecter successfully canceled tick 2024-04-29T20:22:40.398 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f966006e1f0 msgr2=0x7f96600706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f966006e1f0 0x7f96600706b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9670036000 tx=0x7f9670008030 comp rx=0 tx=0).stop 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 10 monclient: shutdown 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 msgr2=0x7f96800aecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 0x7f96800aecc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f967800ec10 tx=0x7f967800c490 comp rx=0 tx=0).stop 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 shutdown_connections 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f966006e1f0 0x7f96600706b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f96800ae050 0x7f96800aecc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96800aa480 0x7f968013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 --2- 172.21.15.64:0/2714876293 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9680072ea0 0x7f96800b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 >> 172.21.15.64:0/2714876293 conn(0x7f96800b9020 msgr2=0x7f96800b9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 shutdown_connections 2024-04-29T20:22:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 -- 172.21.15.64:0/2714876293 wait complete. 2024-04-29T20:22:40.400 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:40.396+0000 7f9684a5c640 1 librados: shutdown 2024-04-29T20:22:40.407 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-04-29T20:22:44.823 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:22:43.971+0000 5dce500 -1 bdev(0x154bab80 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-04-29T20:22:46.122 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:22:46.117+0000 b485640 -1 osd.6 0 waiting for initial osdmap 2024-04-29T20:22:46.589 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:22:46.575+0000 14268640 -1 osd.6 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:47.214 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:47.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.329+0000 7ff08c5ce640 10 monclient: get_monmap_and_config 2024-04-29T20:22:47.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.329+0000 7ff08c5ce640 10 monclient: build_initial_monmap 2024-04-29T20:22:47.331 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.330+0000 7ff08c5ce640 10 monclient: monmap: 2024-04-29T20:22:47.331 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:47.331 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:47.331 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:47.330855+0000 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:47.330855+0000 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:47.332 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.330+0000 7ff08c5ce640 1 Processor -- start 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.330+0000 7ff08c5ce640 1 -- start start 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.330+0000 7ff08c5ce640 10 monclient: init 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff084072ea0 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff084073280 0x7ff084143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff084073280 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff0840aa480 0x7ff0840b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff0840aa480 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0840ab2e0 con 0x7ff084072ea0 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:47.334 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7ff0840aabe0 con 0x7ff084073280 2024-04-29T20:22:47.335 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:47.335 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff0840b34d0 con 0x7ff0840aa480 2024-04-29T20:22:47.335 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:47.335 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.331+0000 7ff08c5ce640 10 monclient(hunting): authenticate will time out at 1516.491229s 2024-04-29T20:22:47.335 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.332+0000 7ff089b69640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff084073280 0x7ff084143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:47.335 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff084073280 0x7ff084143490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60586/0 (socket says 172.21.15.64:60586) 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 1 -- 172.21.15.64:0/609370846 learned_addr learned my addr 172.21.15.64:0/609370846 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): get_auth_request con 0x7ff084073280 auth_method 0 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.333+0000 7ff089b69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 1 -- 172.21.15.64:0/609370846 >> v2:172.21.15.167:3300/0 conn(0x7ff0840aa480 msgr2=0x7ff0840b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 1 --2- 172.21.15.64:0/609370846 >> v2:172.21.15.167:3300/0 conn(0x7ff0840aa480 0x7ff0840b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 1 -- 172.21.15.64:0/609370846 >> v1:172.21.15.167:6789/0 conn(0x7ff084072ea0 legacy=0x7ff0840b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 10 monclient: _finish_hunting 0 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 1 monclient: found mon.noname-a 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:47.337 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.335+0000 7ff089b69640 1 -- 172.21.15.64:0/609370846 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0840b5550 con 0x7ff084073280 2024-04-29T20:22:47.339 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.338+0000 7ff089b69640 1 --2- 172.21.15.64:0/609370846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084073280 0x7ff084143490 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff084143a30 tx=0x7ff07402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1eb68be24a8e8bc server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 1 -- 172.21.15.64:0/609370846 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07402e070 con 0x7ff084073280 2024-04-29T20:22:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: dump: 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: _finish_auth 0 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: _check_auth_tickets 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:47.352 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 1 -- 172.21.15.64:0/609370846 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff074037070 con 0x7ff084073280 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 1 -- 172.21.15.64:0/609370846 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074031a00 con 0x7ff084073280 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff089368640 10 monclient: dump: 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:47.353 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 5 monclient: authenticate success, global_id 4242 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 4 set_mon_vals no callback set 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 10 monclient: shutdown 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 1 -- 172.21.15.64:0/609370846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084073280 msgr2=0x7ff084143490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.343+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/609370846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084073280 0x7ff084143490 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff084143a30 tx=0x7ff07402b880 comp rx=0 tx=0).stop 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.344+0000 7ff08c5ce640 1 -- 172.21.15.64:0/609370846 shutdown_connections 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.344+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/609370846 >> v2:172.21.15.167:3300/0 conn(0x7ff0840aa480 0x7ff0840b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.344+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/609370846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084073280 0x7ff084143490 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.344+0000 7ff08c5ce640 1 -- 172.21.15.64:0/609370846 >> 172.21.15.64:0/609370846 conn(0x7ff0840b9020 msgr2=0x7ff0840b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.344+0000 7ff08c5ce640 1 -- 172.21.15.64:0/609370846 shutdown_connections 2024-04-29T20:22:47.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.344+0000 7ff08c5ce640 1 -- 172.21.15.64:0/609370846 wait complete. 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 10 monclient: build_initial_monmap 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 10 monclient: monmap: 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 1 librados: starting msgr at 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 1 librados: starting objecter 2024-04-29T20:22:47.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 1 Processor -- start 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 1 -- start start 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 1 librados: setting wanted keys 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 1 librados: calling monclient init 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.345+0000 7ff08c5ce640 10 monclient: init 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084072ea0 0x7ff08414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff084072ea0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 0x7ff08406f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff0840aa480 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff0840b0ed0 0x7ff08406f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff0840b0ed0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:47.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840aa8f0 con 0x7ff084072ea0 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840b2770 con 0x7ff0840aa480 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff08414dcc0 con 0x7ff0840b0ed0 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.346+0000 7ff08c5ce640 10 monclient(hunting): authenticate will time out at 1516.506253s 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 0x7ff08406f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 0x7ff08406f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33704/0 (socket says 172.21.15.64:33704) 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 1 -- 172.21.15.64:0/1824535325 learned_addr learned my addr 172.21.15.64:0/1824535325 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): get_auth_request con 0x7ff0840aa480 auth_method 0 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:47.357 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.347+0000 7ff089b69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.348+0000 7ff08a36a640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084072ea0 0x7ff08414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.348+0000 7ff08a36a640 10 monclient(hunting): get_auth_request con 0x7ff084072ea0 auth_method 0 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.348+0000 7ff08a36a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.348+0000 7ff08a36a640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.348+0000 7ff08a36a640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.349+0000 7ff08a36a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.349+0000 7ff08a36a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:47.358 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.349+0000 7ff08a36a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:47.359 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 1 -- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff0840b0ed0 msgr2=0x7ff08406f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff0840b0ed0 0x7ff08406f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 1 -- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084072ea0 msgr2=0x7ff08414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084072ea0 0x7ff08414ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 10 monclient: _finish_hunting 0 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 1 monclient: found mon.noname-b 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.351+0000 7ff089b69640 1 -- 172.21.15.64:0/1824535325 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff084070a00 con 0x7ff0840aa480 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.354+0000 7ff089b69640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 0x7ff08406f450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff07402b8c0 tx=0x7ff0740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.354+0000 7ff07affd640 10 client.?.objecter ms_handle_connect 0x7ff0840aa480 2024-04-29T20:22:47.360 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.354+0000 7ff07affd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074032880 con 0x7ff0840aa480 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: dump: 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:47.363 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: _finish_auth 0 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff074032a20 con 0x7ff0840aa480 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074032e10 con 0x7ff0840aa480 2024-04-29T20:22:47.364 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff07affd640 10 monclient: dump: 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 5 monclient: authenticate success, global_id 4258 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff088b67640 4 set_mon_vals no callback set 2024-04-29T20:22:47.365 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 10 monclient: _renew_subs 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff084071cc0 con 0x7ff0840aa480 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 10 monclient: _renew_subs 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0840717a0 con 0x7ff0840aa480 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.359+0000 7ff08c5ce640 1 librados: init done 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.360+0000 7ff08c5ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.360+0000 7ff08c5ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.360+0000 7ff08c5ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:47.366 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.360+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff084143010 con 0x7ff0840aa480 2024-04-29T20:22:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.384+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7ff07404c060 con 0x7ff0840aa480 2024-04-29T20:22:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.385+0000 7ff07affd640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff06806e240 0x7ff068070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.385+0000 7ff08a36a640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff06806e240 0x7ff068070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.385+0000 7ff08a36a640 10 monclient: get_auth_request con 0x7ff06806e240 auth_method 0 2024-04-29T20:22:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.386+0000 7ff07affd640 10 client.4258.objecter ms_handle_connect 0x7ff06806e240 2024-04-29T20:22:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.386+0000 7ff08a36a640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff06806e240 0x7ff068070700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0800017f0 tx=0x7ff080009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.388+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4017+0+0 (secure 0 0 0) 0x7ff074079950 con 0x7ff0840aa480 2024-04-29T20:22:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.388+0000 7ff07affd640 10 client.4258.objecter ms_dispatch 0x7ff084000b90 osd_map(20..20 src has 1..20) 2024-04-29T20:22:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.388+0000 7ff07affd640 3 client.4258.objecter handle_osd_map got epochs [20,20] > 0 2024-04-29T20:22:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.388+0000 7ff07affd640 3 client.4258.objecter handle_osd_map decoding full epoch 20 2024-04-29T20:22:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.388+0000 7ff07affd640 20 client.4258.objecter dump_active .. 0 homeless 2024-04-29T20:22:47.686 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.684+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff084143010 con 0x7ff0840aa480 2024-04-29T20:22:47.686 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.685+0000 7ff07affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:47.686 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.685+0000 7ff07affd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:47.917 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.916+0000 7ff08c5ce640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:47.918 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.916+0000 7ff08c5ce640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:47.918 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.916+0000 7ff08c5ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:47.918 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:47.916+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff084071970 con 0x7ff0840aa480 2024-04-29T20:22:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.028+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff074077f20 con 0x7ff0840aa480 2024-04-29T20:22:48.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.154+0000 7ff07affd640 1 -- 172.21.15.64:0/1824535325 <== mon.1 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+11429 (secure 0 0 0) 0x7ff084071970 con 0x7ff0840aa480 2024-04-29T20:22:48.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.154+0000 7ff07affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:48.155 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.154+0000 7ff07affd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:48.156 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:48.156 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":20,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:22:47.158637+0000","last_up_change":"2024-04-29T20:22:47.158637+0000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T20:22:41.558205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6801","nonce":2373308255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6803","nonce":2373308255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6807","nonce":2373308255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6805","nonce":2373308255}]},"public_addr":"172.21.15.64:6801/2373308255","cluster_addr":"172.21.15.64:6803/2373308255","heartbeat_back_addr":"172.21.15.64:6807/2373308255","heartbeat_front_addr":"172.21.15.64:6805/2373308255","state":["exists","up"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6827","nonce":3754201859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6829","nonce":3754201859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6833","nonce":3754201859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6831","nonce":3754201859}]},"public_addr":"172.21.15.167:6827/3754201859","cluster_addr":"172.21.15.167:6829/3754201859","heartbeat_back_addr":"172.21.15.167:6833/3754201859","heartbeat_front_addr":"172.21.15.167:6831/3754201859","state":["exists","up"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6803","nonce":3447508053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6805","nonce":3447508053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6809","nonce":3447508053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6807","nonce":3447508053}]},"public_addr":"172.21.15.167:6803/3447508053","cluster_addr":"172.21.15.167:6805/3447508053","heartbeat_back_addr":"172.21.15.167:6809/3447508053","heartbeat_front_addr":"172.21.15.167:6807/3447508053","state":["exists","up"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6819","nonce":3916781155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6821","nonce":3916781155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6825","nonce":3916781155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6823","nonce":3916781155}]},"public_addr":"172.21.15.167:6819/3916781155","cluster_addr":"172.21.15.167:6821/3916781155","heartbeat_back_addr":"172.21.15.167:6825/3916781155","heartbeat_front_addr":"172.21.15.167:6823/3916781155","state":["exists","up"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6811","nonce":2258625350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6813","nonce":2258625350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6817","nonce":2258625350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6815","nonce":2258625350}]},"public_addr":"172.21.15.167:6811/2258625350","cluster_addr":"172.21.15.167:6813/2258625350","heartbeat_back_addr":"172.21.15.167:6817/2258625350","heartbeat_front_addr":"172.21.15.167:6815/2258625350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:38.282652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.557528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.720225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.803337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 10 librados: watch_flush enter 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 10 librados: watch_flush exit 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 20 client.4258.objecter shutdown clearing up homeless session... 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 10 client.4258.objecter successfully canceled tick 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff06806e240 msgr2=0x7ff068070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff06806e240 0x7ff068070700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0800017f0 tx=0x7ff080009040 comp rx=0 tx=0).stop 2024-04-29T20:22:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 10 monclient: shutdown 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 msgr2=0x7ff08406f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 0x7ff08406f450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff07402b8c0 tx=0x7ff0740046c0 comp rx=0 tx=0).stop 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 shutdown_connections 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff06806e240 0x7ff068070700 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff0840b0ed0 0x7ff08406f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0840aa480 0x7ff08406f450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 --2- 172.21.15.64:0/1824535325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff084072ea0 0x7ff08414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.156+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 >> 172.21.15.64:0/1824535325 conn(0x7ff0840b9020 msgr2=0x7ff08413c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.157+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 shutdown_connections 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.157+0000 7ff08c5ce640 1 -- 172.21.15.64:0/1824535325 wait complete. 2024-04-29T20:22:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:48.157+0000 7ff08c5ce640 1 librados: shutdown 2024-04-29T20:22:48.168 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-04-29T20:22:50.313 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:50.218+0000 5dce500 -1 bdev(0x18667150 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 2 2024-04-29T20:22:50.357 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:50.312+0000 5dce500 -1 bdev(0x18667150 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-04-29T20:22:50.493 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:50.441+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:50.766 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:50.702+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:50.778 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:50.763+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:50.834 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:50.777+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:50.972 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:50.562+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 4 2024-04-29T20:22:51.074 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:09.580 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.096 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:50.838+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-04-29T20:22:51.097 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:51.006+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:51.145 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:09.651 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.165 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:09.642 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.175 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:50.972+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.180 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.154+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.214 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:09.720 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.217 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:09.694 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.218 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:51.096+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:51.219 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:50.356+0000 5dce500 -1 bdev(0x18667150 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-04-29T20:22:51.293 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:09.798 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.346 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:51.218+0000 5dce500 -1 bdev(0x154b4c30 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-04-29T20:22:51.440 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.179+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.449 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:09.926 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.487 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.439+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.517 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.452+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.522 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:10.027 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.525 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.486+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.641 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:10.147 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:22:51.698 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.503+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.717 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.517+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:51.816 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:51.700+0000 5dce500 -1 bdev(0x154eb4c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-04-29T20:22:52.740 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:52.735+0000 bc99640 -1 osd.0 0 waiting for initial osdmap 2024-04-29T20:22:52.936 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:52.930+0000 b485640 -1 osd.1 0 waiting for initial osdmap 2024-04-29T20:22:53.139 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:53.132+0000 b485640 -1 osd.3 0 waiting for initial osdmap 2024-04-29T20:22:53.318 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:22:53.295+0000 14268640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:53.462 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:22:53.446+0000 14268640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:53.676 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:22:53.661+0000 14268640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T20:22:54.976 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:55.069 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.067+0000 7f0fb15d0640 10 monclient: get_monmap_and_config 2024-04-29T20:22:55.069 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.067+0000 7f0fb15d0640 10 monclient: build_initial_monmap 2024-04-29T20:22:55.069 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.067+0000 7f0fb15d0640 10 monclient: monmap: 2024-04-29T20:22:55.069 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:55.069 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.069 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:22:55.068827+0000 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:22:55.068827+0000 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:55.070 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.068+0000 7f0fb15d0640 1 Processor -- start 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.068+0000 7f0fb15d0640 1 -- start start 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.068+0000 7f0fb15d0640 10 monclient: init 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fac072ea0 addr v1:172.21.15.64:6789/0 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fac073280 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac143490 0x7f0fac0aa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0fac143490 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fac0b2730 con 0x7f0fac072ea0 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fac0ab2e0 con 0x7f0fac073280 2024-04-29T20:22:55.072 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f0fac143900 con 0x7f0fac143490 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0fb15d0640 10 monclient(hunting): authenticate will time out at 1524.228908s 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.069+0000 7f0faaffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0fac072ea0 0x7f0fac0b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51796/0 (socket says 172.21.15.64:51796) 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.070+0000 7f0faaffd640 1 -- 172.21.15.64:0/1868059527 learned_addr learned my addr 172.21.15.64:0/1868059527 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.070+0000 7f0fab7fe640 1 --2- 172.21.15.64:0/1868059527 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac143490 0x7f0fac0aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.070+0000 7f0fab7fe640 10 monclient(hunting): get_auth_request con 0x7f0fac143490 auth_method 0 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.070+0000 7f0fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.070+0000 7f0fab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.073 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.070+0000 7f0fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.074 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.071+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.071+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.071+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483874655 0 0) 0x7f0fa0001110 con 0x7f0fac072ea0 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): _negotiate 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): my global_id is 4251 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f94003520 con 0x7f0fac072ea0 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257063201 0 0) 0x7f0fac0ab2e0 con 0x7f0fac073280 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): _negotiate 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 10 monclient(hunting): my global_id is 4264 2024-04-29T20:22:55.075 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.072+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa0001110 con 0x7f0fac073280 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 1 -- 172.21.15.64:0/1868059527 >> v1:172.21.15.167:6789/0 conn(0x7f0fac073280 legacy=0x7f0fac0b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 1 -- 172.21.15.64:0/1868059527 >> v1:172.21.15.64:6789/0 conn(0x7f0fac072ea0 legacy=0x7f0fac0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 1 monclient: found mon.noname-b 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:22:55.076 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.073+0000 7f0fab7fe640 1 -- 172.21.15.64:0/1868059527 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fac0b5550 con 0x7f0fac143490 2024-04-29T20:22:55.077 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.075+0000 7f0fab7fe640 1 --2- 172.21.15.64:0/1868059527 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac143490 0x7f0fac0aa480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009dc0 tx=0x7f0fa402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9962670db15aacf server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4039070 con 0x7f0fac143490 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: dump: 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.081 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: _finish_auth 0 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa402cc40 con 0x7f0fac143490 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 5 monclient: authenticate success, global_id 4211 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:55.082 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 4 set_mon_vals no callback set 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 1 -- 172.21.15.64:0/1868059527 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4034440 con 0x7f0fac143490 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fa9ffb640 10 monclient: dump: 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.083 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 10 monclient: shutdown 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 -- 172.21.15.64:0/1868059527 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac143490 msgr2=0x7f0fac0aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/1868059527 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac143490 0x7f0fac0aa480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009dc0 tx=0x7f0fa402be90 comp rx=0 tx=0).stop 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 -- 172.21.15.64:0/1868059527 shutdown_connections 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/1868059527 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac143490 0x7f0fac0aa480 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 -- 172.21.15.64:0/1868059527 >> 172.21.15.64:0/1868059527 conn(0x7f0fac0b9020 msgr2=0x7f0fac0b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:55.087 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 -- 172.21.15.64:0/1868059527 shutdown_connections 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.080+0000 7f0fb15d0640 1 -- 172.21.15.64:0/1868059527 wait complete. 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.081+0000 7f0fb15d0640 10 monclient: build_initial_monmap 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.081+0000 7f0fb15d0640 10 monclient: monmap: 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.081+0000 7f0fb15d0640 1 librados: starting msgr at 2024-04-29T20:22:55.088 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.081+0000 7f0fb15d0640 1 librados: starting objecter 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 Processor -- start 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 -- start start 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 librados: setting wanted keys 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 librados: calling monclient init 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient: init 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 0x7f0fac0b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0fac072ea0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac143490 0x7f0fac13a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0fac143490 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac13a640 0x7f0fac0ad120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0fac13a640 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.089 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac0b34d0 con 0x7f0fac143490 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac14eee0 con 0x7f0fac072ea0 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f0fac14f060 con 0x7f0fac13a640 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.082+0000 7f0fb15d0640 10 monclient(hunting): authenticate will time out at 1524.242436s 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 0x7f0fac0b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0fab7fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac13a640 0x7f0fac0ad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faa7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac143490 0x7f0fac13a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 0x7f0fac0b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33346/0 (socket says 172.21.15.64:33346) 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 1 -- 172.21.15.64:0/3101517308 learned_addr learned my addr 172.21.15.64:0/3101517308 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 10 monclient(hunting): get_auth_request con 0x7f0fac072ea0 auth_method 0 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.090 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0fab7fe640 10 monclient(hunting): get_auth_request con 0x7f0fac13a640 auth_method 0 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0fab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faa7fc640 10 monclient(hunting): get_auth_request con 0x7f0fac143490 auth_method 0 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faa7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.083+0000 7f0faa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0faa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.092 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0faa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.092 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.084+0000 7f0faa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.093 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 10 monclient(hunting): handle_auth_done global_id 4267 payload 824 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 1 -- 172.21.15.64:0/3101517308 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac13a640 msgr2=0x7f0fac0ad120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac13a640 0x7f0fac0ad120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 1 -- 172.21.15.64:0/3101517308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac143490 msgr2=0x7f0fac13a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac143490 0x7f0fac13a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 10 monclient: _finish_hunting 0 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 1 monclient: found mon.noname-b 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.087+0000 7f0faaffd640 1 -- 172.21.15.64:0/3101517308 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fac0b1650 con 0x7f0fac072ea0 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.089+0000 7f0faaffd640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 0x7f0fac0b03a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000c6b0 tx=0x7f0fa000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.089+0000 7f0f937fe640 10 client.?.objecter ms_handle_connect 0x7f0fac072ea0 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.089+0000 7f0f937fe640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:55.094 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0017070 con 0x7f0fac072ea0 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 10 monclient: dump: 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.095 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 10 monclient: _finish_auth 0 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 10 monclient: _check_auth_tickets 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0f937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 5 monclient: authenticate success, global_id 4267 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 10 monclient: _renew_subs 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fac144980 con 0x7f0fac072ea0 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 10 monclient: _renew_subs 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fac13aaf0 con 0x7f0fac072ea0 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.091+0000 7f0fb15d0640 1 librados: init done 2024-04-29T20:22:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.092+0000 7f0fb15d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.092+0000 7f0fb15d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.092+0000 7f0fb15d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.092+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fac0b0dc0 con 0x7f0fac072ea0 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa0005ab0 con 0x7f0fac072ea0 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0f937fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0021060 con 0x7f0fac072ea0 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0f937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0f937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0f937fe640 10 monclient: dump: 2024-04-29T20:22:55.353 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr: 2024-04-29T20:22:55.354 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.351+0000 7f0fa97fa640 4 set_mon_vals no callback set 2024-04-29T20:22:55.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.352+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0fa001a070 con 0x7f0fac072ea0 2024-04-29T20:22:55.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.353+0000 7f0f937fe640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f0f9806e280 0x7f0f98070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.353+0000 7f0faa7fc640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f0f9806e280 0x7f0f98070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.353+0000 7f0faa7fc640 10 monclient: get_auth_request con 0x7f0f9806e280 auth_method 0 2024-04-29T20:22:55.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.353+0000 7f0faa7fc640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f0f9806e280 0x7f0f98070740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0fac14ea20 tx=0x7f0f9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:55.355 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.353+0000 7f0f937fe640 10 client.4267.objecter ms_handle_connect 0x7f0f9806e280 2024-04-29T20:22:55.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.355+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4857+0+0 (secure 0 0 0) 0x7f0fa005fc20 con 0x7f0fac072ea0 2024-04-29T20:22:55.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.355+0000 7f0f937fe640 10 client.4267.objecter ms_dispatch 0x7f0fac000b90 osd_map(22..22 src has 1..22) 2024-04-29T20:22:55.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.355+0000 7f0f937fe640 3 client.4267.objecter handle_osd_map got epochs [22,22] > 0 2024-04-29T20:22:55.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.355+0000 7f0f937fe640 3 client.4267.objecter handle_osd_map decoding full epoch 22 2024-04-29T20:22:55.356 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.355+0000 7f0f937fe640 20 client.4267.objecter dump_active .. 0 homeless 2024-04-29T20:22:55.368 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.367+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0fac0b0dc0 con 0x7f0fac072ea0 2024-04-29T20:22:55.368 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.367+0000 7f0f937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:55.368 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.367+0000 7f0f937fe640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:55.590 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.588+0000 7f0fb15d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:55.590 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.588+0000 7f0fb15d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:55.590 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.588+0000 7f0fb15d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.590 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.588+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fac0b0ac0 con 0x7f0fac072ea0 2024-04-29T20:22:55.614 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.613+0000 7f0f937fe640 1 -- 172.21.15.64:0/3101517308 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+12884 (secure 0 0 0) 0x7f0fac0b2730 con 0x7f0fac072ea0 2024-04-29T20:22:55.614 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.613+0000 7f0f937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:55.614 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.613+0000 7f0f937fe640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:55.615 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-29T20:22:55.615 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":23,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:22:55.409329+0000","last_up_change":"2024-04-29T20:22:54.303806+0000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T20:22:41.558205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6810","nonce":2796311601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6815","nonce":2796311601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6823","nonce":2796311601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6821","nonce":2796311601}]},"public_addr":"172.21.15.64:6810/2796311601","cluster_addr":"172.21.15.64:6815/2796311601","heartbeat_back_addr":"172.21.15.64:6823/2796311601","heartbeat_front_addr":"172.21.15.64:6821/2796311601","state":["exists","up"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6811","nonce":3421706551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6814","nonce":3421706551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6820","nonce":3421706551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6817","nonce":3421706551}]},"public_addr":"172.21.15.64:6811/3421706551","cluster_addr":"172.21.15.64:6814/3421706551","heartbeat_back_addr":"172.21.15.64:6820/3421706551","heartbeat_front_addr":"172.21.15.64:6817/3421706551","state":["exists","up"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6801","nonce":2373308255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6803","nonce":2373308255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6807","nonce":2373308255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6805","nonce":2373308255}]},"public_addr":"172.21.15.64:6801/2373308255","cluster_addr":"172.21.15.64:6803/2373308255","heartbeat_back_addr":"172.21.15.64:6807/2373308255","heartbeat_front_addr":"172.21.15.64:6805/2373308255","state":["exists","up"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6825","nonce":471606907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6827","nonce":471606907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6831","nonce":471606907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6829","nonce":471606907}]},"public_addr":"172.21.15.64:6825/471606907","cluster_addr":"172.21.15.64:6827/471606907","heartbeat_back_addr":"172.21.15.64:6831/471606907","heartbeat_front_addr":"172.21.15.64:6829/471606907","state":["exists","up"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6827","nonce":3754201859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6829","nonce":3754201859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6833","nonce":3754201859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6831","nonce":3754201859}]},"public_addr":"172.21.15.167:6827/3754201859","cluster_addr":"172.21.15.167:6829/3754201859","heartbeat_back_addr":"172.21.15.167:6833/3754201859","heartbeat_front_addr":"172.21.15.167:6831/3754201859","state":["exists","up"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6803","nonce":3447508053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6805","nonce":3447508053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6809","nonce":3447508053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6807","nonce":3447508053}]},"public_addr":"172.21.15.167:6803/3447508053","cluster_addr":"172.21.15.167:6805/3447508053","heartbeat_back_addr":"172.21.15.167:6809/3447508053","heartbeat_front_addr":"172.21.15.167:6807/3447508053","state":["exists","up"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6819","nonce":3916781155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6821","nonce":3916781155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6825","nonce":3916781155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6823","nonce":3916781155}]},"public_addr":"172.21.15.167:6819/3916781155","cluster_addr":"172.21.15.167:6821/3916781155","heartbeat_back_addr":"172.21.15.167:6825/3916781155","heartbeat_front_addr":"172.21.15.167:6823/3916781155","state":["exists","up"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6811","nonce":2258625350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6813","nonce":2258625350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6817","nonce":2258625350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6815","nonce":2258625350}]},"public_addr":"172.21.15.167:6811/2258625350","cluster_addr":"172.21.15.167:6813/2258625350","heartbeat_back_addr":"172.21.15.167:6817/2258625350","heartbeat_front_addr":"172.21.15.167:6815/2258625350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:40.154276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.540513+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:38.282652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.442559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.557528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.720225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:35.815755+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.803337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 10 librados: watch_flush enter 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 10 librados: watch_flush exit 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 20 client.4267.objecter shutdown clearing up homeless session... 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 10 client.4267.objecter successfully canceled tick 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f0f9806e280 msgr2=0x7f0f98070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f0f9806e280 0x7f0f98070740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0fac14ea20 tx=0x7f0f9c009040 comp rx=0 tx=0).stop 2024-04-29T20:22:55.617 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 10 monclient: shutdown 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 msgr2=0x7f0fac0b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 0x7f0fac0b03a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000c6b0 tx=0x7f0fa000cb70 comp rx=0 tx=0).stop 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 shutdown_connections 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f0f9806e280 0x7f0f98070740 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f0fac13a640 0x7f0fac0ad120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac143490 0x7f0fac13a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 --2- 172.21.15.64:0/3101517308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fac072ea0 0x7f0fac0b03a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 >> 172.21.15.64:0/3101517308 conn(0x7f0fac0b9020 msgr2=0x7f0fac0aad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 shutdown_connections 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 -- 172.21.15.64:0/3101517308 wait complete. 2024-04-29T20:22:55.618 INFO:teuthology.misc.health.smithi064.stderr:2024-04-29T20:22:55.615+0000 7f0fb15d0640 1 librados: shutdown 2024-04-29T20:22:55.630 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-29T20:22:55.630 INFO:tasks.ceph:Creating RBD pool 2024-04-29T20:22:55.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-29T20:22:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.730+0000 7fa2c0bc7640 10 monclient: get_monmap_and_config 2024-04-29T20:22:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.730+0000 7fa2c0bc7640 10 monclient: build_initial_monmap 2024-04-29T20:22:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.730+0000 7fa2c0bc7640 10 monclient: monmap: 2024-04-29T20:22:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:22:55.730955+0000 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:22:55.730955+0000 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:55.732 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.730+0000 7fa2c0bc7640 1 Processor -- start 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.730+0000 7fa2c0bc7640 1 -- start start 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.730+0000 7fa2c0bc7640 10 monclient: init 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa2bc0b22a0 0x7fa2bc146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): picked mon.noname-c con 0x7fa2bc0b22a0 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2bc0b2c20 addr v1:172.21.15.64:6789/0 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa2bc0aa6a0 0x7fa2bc14ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc0aa6a0 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2bc0ac1f0 con 0x7fa2bc0b2c20 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fa2bc0b33d0 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fa2bc0ac4d0 con 0x7fa2bc0b22a0 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.731+0000 7fa2c0bc7640 10 monclient(hunting): authenticate will time out at 1524.891164s 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa2bc0aa6a0 0x7fa2bc14ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2ba59c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa2bc0b22a0 0x7fa2bc146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa2bc0aa6a0 0x7fa2bc14ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34342/0 (socket says 172.21.15.64:34342) 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 1 -- 172.21.15.64:0/1589415086 learned_addr learned my addr 172.21.15.64:0/1589415086 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 10 monclient(hunting): get_auth_request con 0x7fa2bc0aa6a0 auth_method 0 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2ba59c640 10 monclient(hunting): get_auth_request con 0x7fa2bc0b22a0 auth_method 0 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.732+0000 7fa2ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.733+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.733+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.733+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.733+0000 7fa2bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.733+0000 7fa2bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.733+0000 7fa2bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 1 -- 172.21.15.64:0/1589415086 >> v2:172.21.15.64:3300/0 conn(0x7fa2bc0aa6a0 msgr2=0x7fa2bc14ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 1 --2- 172.21.15.64:0/1589415086 >> v2:172.21.15.64:3300/0 conn(0x7fa2bc0aa6a0 0x7fa2bc14ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 1 -- 172.21.15.64:0/1589415086 >> v1:172.21.15.64:6789/0 conn(0x7fa2bc0b2c20 legacy=0x7fa2bc0b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 10 monclient: _finish_hunting 0 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 1 monclient: found mon.noname-c 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.734+0000 7fa2ba59c640 1 -- 172.21.15.64:0/1589415086 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc0b3040 con 0x7fa2bc0b22a0 2024-04-29T20:22:55.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.736+0000 7fa2ba59c640 1 --2- 172.21.15.64:0/1589415086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc146a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc06d6e0 tx=0x7fa2ac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5cfb6671898245 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:55.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 1 -- 172.21.15.64:0/1589415086 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac0315a0 con 0x7fa2bc0b22a0 2024-04-29T20:22:55.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: dump: 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: _finish_auth 0 2024-04-29T20:22:55.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: _check_auth_tickets 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 1 -- 172.21.15.64:0/1589415086 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ac037070 con 0x7fa2bc0b22a0 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2c0bc7640 5 monclient: authenticate success, global_id 4270 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2c0bc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2c0bc7640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2c0bc7640 4 set_mon_vals no callback set 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 1 -- 172.21.15.64:0/1589415086 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac031960 con 0x7fa2bc0b22a0 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.740+0000 7fa2b959a640 10 monclient: dump: 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.743 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.744 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.744 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.744 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.744 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.744 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.744 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 10 monclient: shutdown 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/1589415086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0b22a0 msgr2=0x7fa2bc146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/1589415086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc146a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc06d6e0 tx=0x7fa2ac02b840 comp rx=0 tx=0).stop 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/1589415086 shutdown_connections 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/1589415086 >> v2:172.21.15.64:3300/0 conn(0x7fa2bc0aa6a0 0x7fa2bc14ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/1589415086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc146a40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/1589415086 >> 172.21.15.64:0/1589415086 conn(0x7fa2bc06c940 msgr2=0x7fa2bc06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/1589415086 shutdown_connections 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.741+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/1589415086 wait complete. 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 10 monclient: build_initial_monmap 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 10 monclient: monmap: 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:55.748 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 1 librados: starting msgr at 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 1 librados: starting objecter 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 1 Processor -- start 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 1 -- start start 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 1 librados: setting wanted keys 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 1 librados: calling monclient init 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.742+0000 7fa2c0bc7640 10 monclient: init 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0aa6a0 0x7fa2bc0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc0aa6a0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc1d89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc0b22a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:55.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa2bc0b2c20 0x7fa2bc1d8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:55.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): picked mon.noname-c con 0x7fa2bc0b2c20 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:55.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc0aab30 con 0x7fa2bc0b22a0 2024-04-29T20:22:55.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc0b2890 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc0aacb0 con 0x7fa2bc0b2c20 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2c0bc7640 10 monclient(hunting): authenticate will time out at 1524.902629s 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc1d89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2bad9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa2bc0b2c20 0x7fa2bc1d8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.743+0000 7fa2ba59c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0aa6a0 0x7fa2bc0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:55.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa2bc0b2c20 0x7fa2bc1d8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36540/0 (socket says 172.21.15.64:36540) 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 1 -- 172.21.15.64:0/3698380184 learned_addr learned my addr 172.21.15.64:0/3698380184 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): get_auth_request con 0x7fa2bc0b2c20 auth_method 0 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): get_auth_request con 0x7fa2bc0b22a0 auth_method 0 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): get_auth_request con 0x7fa2bc0aa6a0 auth_method 0 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:55.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2b9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:55.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.744+0000 7fa2bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-04-29T20:22:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 1 -- 172.21.15.64:0/3698380184 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa2bc0b2c20 msgr2=0x7fa2bc1d8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa2bc0b2c20 0x7fa2bc1d8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 1 -- 172.21.15.64:0/3698380184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2bc0b22a0 msgr2=0x7fa2bc1d89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc1d89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 10 monclient: _finish_hunting 0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 1 monclient: found mon.noname-b 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.747+0000 7fa2ba59c640 1 -- 172.21.15.64:0/3698380184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc0ad430 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.749+0000 7fa2ba59c640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0aa6a0 0x7fa2bc0b0d10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0b3550 tx=0x7fa2ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.749+0000 7fa2aaffd640 10 client.?.objecter ms_handle_connect 0x7fa2bc0aa6a0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.749+0000 7fa2aaffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac032840 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: dump: 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:55.756 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.757 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.757 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.757 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.757 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: _finish_auth 0 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 5 monclient: authenticate success, global_id 4273 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ac0329e0 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 10 monclient: _renew_subs 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac032db0 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc121290 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2b8d99640 4 set_mon_vals no callback set 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2aaffd640 10 monclient: dump: 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:55.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 10 monclient: _renew_subs 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc0b1430 con 0x7fa2bc0aa6a0 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.752+0000 7fa2c0bc7640 1 librados: init done 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.754+0000 7fa2c0bc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.754+0000 7fa2c0bc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.754+0000 7fa2c0bc7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:55.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:55.754+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2bc04fce0 con 0x7fa2bc0aa6a0 2024-04-29T20:22:56.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.012+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa2ac04d3f0 con 0x7fa2bc0aa6a0 2024-04-29T20:22:56.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.012+0000 7fa2aaffd640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa29c06e2d0 0x7fa29c070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:56.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.012+0000 7fa2b9d9b640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa29c06e2d0 0x7fa29c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:56.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.012+0000 7fa2b9d9b640 10 monclient: get_auth_request con 0x7fa29c06e2d0 auth_method 0 2024-04-29T20:22:56.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.013+0000 7fa2aaffd640 10 client.4273.objecter ms_handle_connect 0x7fa29c06e2d0 2024-04-29T20:22:56.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.013+0000 7fa2b9d9b640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa29c06e2d0 0x7fa29c070790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0b9b40 tx=0x7fa2b0009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:56.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.017+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 4857+0+0 (secure 0 0 0) 0x7fa2ac02f020 con 0x7fa2bc0aa6a0 2024-04-29T20:22:56.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.017+0000 7fa2aaffd640 10 client.4273.objecter ms_dispatch 0x7fa2bc000b90 osd_map(23..23 src has 1..23) 2024-04-29T20:22:56.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.017+0000 7fa2aaffd640 3 client.4273.objecter handle_osd_map got epochs [23,23] > 0 2024-04-29T20:22:56.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.017+0000 7fa2aaffd640 3 client.4273.objecter handle_osd_map decoding full epoch 23 2024-04-29T20:22:56.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.017+0000 7fa2aaffd640 20 client.4273.objecter dump_active .. 0 homeless 2024-04-29T20:22:56.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.027+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2ac040070 con 0x7fa2bc0aa6a0 2024-04-29T20:22:56.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.027+0000 7fa2aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:56.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.027+0000 7fa2aaffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:56.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.215+0000 7fa2c0bc7640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-29T20:22:56.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.215+0000 7fa2c0bc7640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-29T20:22:56.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.215+0000 7fa2c0bc7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:56.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.215+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fa2bc1d9470 con 0x7fa2bc0aa6a0 2024-04-29T20:22:56.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.672+0000 7fa2aaffd640 1 -- 172.21.15.64:0/3698380184 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v24) ==== 109+0+0 (secure 0 0 0) 0x7fa2ac07b580 con 0x7fa2bc0aa6a0 2024-04-29T20:22:56.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.672+0000 7fa2aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-29T20:22:56.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.672+0000 7fa2aaffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-29T20:22:56.674 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 10 librados: watch_flush enter 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 10 librados: watch_flush exit 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 20 client.4273.objecter shutdown clearing up homeless session... 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 10 client.4273.objecter successfully canceled tick 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa29c06e2d0 msgr2=0x7fa29c070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa29c06e2d0 0x7fa29c070790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0b9b40 tx=0x7fa2b0009060 comp rx=0 tx=0).stop 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 10 monclient: shutdown 2024-04-29T20:22:56.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0aa6a0 msgr2=0x7fa2bc0b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0aa6a0 0x7fa2bc0b0d10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0b3550 tx=0x7fa2ac0046c0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 shutdown_connections 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa29c06e2d0 0x7fa29c070790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa2bc0b2c20 0x7fa2bc1d8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2bc0b22a0 0x7fa2bc1d89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 --2- 172.21.15.64:0/3698380184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2bc0aa6a0 0x7fa2bc0b0d10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 >> 172.21.15.64:0/3698380184 conn(0x7fa2bc06c940 msgr2=0x7fa2bc0b73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 shutdown_connections 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 -- 172.21.15.64:0/3698380184 wait complete. 2024-04-29T20:22:56.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.675+0000 7fa2c0bc7640 1 librados: shutdown 2024-04-29T20:22:56.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-29T20:22:56.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.796+0000 7f81a32c1640 10 monclient: get_monmap_and_config 2024-04-29T20:22:56.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.796+0000 7f81a32c1640 10 monclient: build_initial_monmap 2024-04-29T20:22:56.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.796+0000 7f81a32c1640 10 monclient: monmap: 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:22:56.797092+0000 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:22:56.797092+0000 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:56.798 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.796+0000 7f81a32c1640 1 Processor -- start 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.797+0000 7f81a32c1640 1 -- start start 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.797+0000 7f81a32c1640 10 monclient: init 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f819c06cdb0 0x7f819c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): picked mon.noname-a con 0x7f819c06cdb0 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:56.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c0b3270 0x7f819c140d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): picked mon.noname-b con 0x7f819c0b3270 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f819c146a50 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819c0abff0 con 0x7f819c146a50 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f819c06d190 con 0x7f819c06cdb0 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f819c0ab5d0 con 0x7f819c0b3270 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a32c1640 10 monclient(hunting): authenticate will time out at 1525.957740s 2024-04-29T20:22:56.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a105d640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f819c06cdb0 0x7f819c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:56.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a185e640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f819c146a50 0x7f819c146ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40554/0 (socket says 172.21.15.64:40554) 2024-04-29T20:22:56.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.798+0000 7f81a185e640 1 -- 172.21.15.64:0/2627937915 learned_addr learned my addr 172.21.15.64:0/2627937915 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:22:56.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): get_auth_request con 0x7f819c06cdb0 auth_method 0 2024-04-29T20:22:56.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:56.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:56.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:56.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:56.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:56.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.799+0000 7f81a105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f8193fff640 1 -- 172.21.15.64:0/2627937915 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461631593 0 0) 0x7f819c0abff0 con 0x7f819c146a50 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f8193fff640 10 monclient(hunting): _negotiate 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f8193fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f8193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f8193fff640 10 monclient(hunting): my global_id is 4276 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f8193fff640 1 -- 172.21.15.64:0/2627937915 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8188003520 con 0x7f819c146a50 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 1 -- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c0b3270 msgr2=0x7f819c140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 1 --2- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c0b3270 0x7f819c140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 1 -- 172.21.15.64:0/2627937915 >> v1:172.21.15.167:6789/0 conn(0x7f819c146a50 legacy=0x7f819c146ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 10 monclient: _finish_hunting 0 2024-04-29T20:22:56.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 1 monclient: found mon.noname-a 2024-04-29T20:22:56.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:56.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:56.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.801+0000 7f81a105d640 1 -- 172.21.15.64:0/2627937915 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c0ac2d0 con 0x7f819c06cdb0 2024-04-29T20:22:56.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.803+0000 7f81a105d640 1 --2- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c06cdb0 0x7f819c1407c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f81980097a0 tx=0x7f819802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfead9c5e78a887 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 1 -- 172.21.15.64:0/2627937915 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81980315a0 con 0x7f819c06cdb0 2024-04-29T20:22:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 10 monclient: dump: 2024-04-29T20:22:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 10 monclient: _finish_auth 0 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 10 monclient: _check_auth_tickets 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f8193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f81a32c1640 5 monclient: authenticate success, global_id 4263 2024-04-29T20:22:56.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.806+0000 7f81a32c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f8193fff640 1 -- 172.21.15.64:0/2627937915 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8198037070 con 0x7f819c06cdb0 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f8193fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f8193fff640 1 -- 172.21.15.64:0/2627937915 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198031940 con 0x7f819c06cdb0 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 4 set_mon_vals no callback set 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f8193fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:56.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f8193fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f8193fff640 10 monclient: dump: 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:56.810 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:56.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 10 monclient: shutdown 2024-04-29T20:22:56.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 1 -- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c06cdb0 msgr2=0x7f819c1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c06cdb0 0x7f819c1407c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f81980097a0 tx=0x7f819802b840 comp rx=0 tx=0).stop 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 1 -- 172.21.15.64:0/2627937915 shutdown_connections 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c0b3270 0x7f819c140d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.808+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2627937915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c06cdb0 0x7f819c1407c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.809+0000 7f81a32c1640 1 -- 172.21.15.64:0/2627937915 >> 172.21.15.64:0/2627937915 conn(0x7f819c0b9030 msgr2=0x7f819c0b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.809+0000 7f81a32c1640 1 -- 172.21.15.64:0/2627937915 shutdown_connections 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.809+0000 7f81a32c1640 1 -- 172.21.15.64:0/2627937915 wait complete. 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 10 monclient: build_initial_monmap 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 10 monclient: monmap: 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 1 librados: starting msgr at 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 1 librados: starting objecter 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 1 Processor -- start 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 1 -- start start 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 1 librados: setting wanted keys 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 1 librados: calling monclient init 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.810+0000 7f81a32c1640 10 monclient: init 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f819c06cdb0 0x7f819c0b87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): picked mon.noname-b con 0x7f819c06cdb0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 0x7f819c143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): picked mon.noname-a con 0x7f819c0b3270 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c146a50 0x7f819c1434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): picked mon.noname-c con 0x7f819c146a50 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f819c0aace0 con 0x7f819c0b3270 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1475d0 con 0x7f819c06cdb0 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f819c0aae60 con 0x7f819c146a50 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a32c1640 10 monclient(hunting): authenticate will time out at 1525.970753s 2024-04-29T20:22:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a085c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 0x7f819c143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.811+0000 7f81a105d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f819c06cdb0 0x7f819c0b87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:56.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 0x7f819c143fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34374/0 (socket says 172.21.15.64:34374) 2024-04-29T20:22:56.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 1 -- 172.21.15.64:0/2930035644 learned_addr learned my addr 172.21.15.64:0/2930035644 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:56.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): get_auth_request con 0x7f819c0b3270 auth_method 0 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): get_auth_request con 0x7f819c06cdb0 auth_method 0 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a085c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.812+0000 7f81a105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a185e640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c146a50 0x7f819c1434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:56.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 1 -- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c146a50 msgr2=0x7f819c1434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c146a50 0x7f819c1434a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 1 -- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 msgr2=0x7f819c143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 0x7f819c143fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 10 monclient: _finish_hunting 0 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 1 monclient: found mon.noname-b 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a105d640 1 -- 172.21.15.64:0/2930035644 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c1d8530 con 0x7f819c06cdb0 2024-04-29T20:22:56.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.815+0000 7f81a085c640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 0x7f819c143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-29T20:22:56.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.818+0000 7f81a105d640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f819c06cdb0 0x7f819c0b87a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8198033000 tx=0x7f81980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:56.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.818+0000 7f8191ffb640 10 client.?.objecter ms_handle_connect 0x7f819c06cdb0 2024-04-29T20:22:56.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.818+0000 7f8191ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:56.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81980317a0 con 0x7f819c06cdb0 2024-04-29T20:22:56.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:56.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:56.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: dump: 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: _finish_auth 0 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8198031940 con 0x7f819c06cdb0 2024-04-29T20:22:56.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f81a32c1640 5 monclient: authenticate success, global_id 4279 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f81a32c1640 10 monclient: _renew_subs 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f81a32c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819c1d87c0 con 0x7f819c06cdb0 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198031d30 con 0x7f819c06cdb0 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f8191ffb640 10 monclient: dump: 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:56.825 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.821+0000 7f81937fe640 4 set_mon_vals no callback set 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.822+0000 7f81a32c1640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.822+0000 7f81a32c1640 10 monclient: _renew_subs 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.822+0000 7f81a32c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.822+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f819c1d8ce0 con 0x7f819c06cdb0 2024-04-29T20:22:56.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.822+0000 7f81a32c1640 1 librados: init done 2024-04-29T20:22:56.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.823+0000 7f81a32c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:56.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.823+0000 7f81a32c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:56.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.823+0000 7f81a32c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:56.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:56.823+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f819c04fce0 con 0x7f819c06cdb0 2024-04-29T20:22:57.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.098+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8198051480 con 0x7f819c06cdb0 2024-04-29T20:22:57.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.098+0000 7f8191ffb640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f818006e2d0 0x7f8180070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:57.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.098+0000 7f81a085c640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f818006e2d0 0x7f8180070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:57.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.098+0000 7f81a085c640 10 monclient: get_auth_request con 0x7f818006e2d0 auth_method 0 2024-04-29T20:22:57.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.099+0000 7f8191ffb640 10 client.4279.objecter ms_handle_connect 0x7f818006e2d0 2024-04-29T20:22:57.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.099+0000 7f81a085c640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f818006e2d0 0x7f8180070790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f819c142fb0 tx=0x7f818c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:57.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.102+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5199+0+0 (secure 0 0 0) 0x7f819802f060 con 0x7f819c06cdb0 2024-04-29T20:22:57.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.102+0000 7f8191ffb640 10 client.4279.objecter ms_dispatch 0x7f819c000b90 osd_map(24..24 src has 1..24) 2024-04-29T20:22:57.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.102+0000 7f8191ffb640 3 client.4279.objecter handle_osd_map got epochs [24,24] > 0 2024-04-29T20:22:57.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.102+0000 7f8191ffb640 3 client.4279.objecter handle_osd_map decoding full epoch 24 2024-04-29T20:22:57.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.102+0000 7f8191ffb640 20 client.4279.objecter dump_active .. 0 homeless 2024-04-29T20:22:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.115+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f819804c020 con 0x7f819c06cdb0 2024-04-29T20:22:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.115+0000 7f8191ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.115+0000 7f8191ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.331+0000 7f81a32c1640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-29T20:22:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.331+0000 7f81a32c1640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-29T20:22:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.331+0000 7f81a32c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.331+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f819c071f90 con 0x7f819c06cdb0 2024-04-29T20:22:57.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.866+0000 7f8191ffb640 1 -- 172.21.15.64:0/2930035644 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v25) ==== 173+0+0 (secure 0 0 0) 0x7f819807cb20 con 0x7f819c06cdb0 2024-04-29T20:22:57.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.866+0000 7f8191ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-29T20:22:57.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.866+0000 7f8191ffb640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-29T20:22:57.868 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 10 librados: watch_flush enter 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 10 librados: watch_flush exit 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 20 client.4279.objecter shutdown clearing up homeless session... 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 10 client.4279.objecter successfully canceled tick 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f818006e2d0 msgr2=0x7f8180070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f818006e2d0 0x7f8180070790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f819c142fb0 tx=0x7f818c009190 comp rx=0 tx=0).stop 2024-04-29T20:22:57.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 10 monclient: shutdown 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f819c06cdb0 msgr2=0x7f819c0b87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f819c06cdb0 0x7f819c0b87a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8198033000 tx=0x7f81980046c0 comp rx=0 tx=0).stop 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 shutdown_connections 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f818006e2d0 0x7f8180070790 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f819c146a50 0x7f819c1434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f819c0b3270 0x7f819c143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 --2- 172.21.15.64:0/2930035644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f819c06cdb0 0x7f819c0b87a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 >> 172.21.15.64:0/2930035644 conn(0x7f819c0b9030 msgr2=0x7f819c0b9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 shutdown_connections 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 -- 172.21.15.64:0/2930035644 wait complete. 2024-04-29T20:22:57.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:57.868+0000 7f81a32c1640 1 librados: shutdown 2024-04-29T20:22:57.881 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-29T20:22:57.881 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-29T20:22:57.881 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-29T20:22:57.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-29T20:22:57.886 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-29T20:22:58.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.003+0000 7f38074a5640 10 monclient: get_monmap_and_config 2024-04-29T20:22:58.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.003+0000 7f38074a5640 10 monclient: build_initial_monmap 2024-04-29T20:22:58.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.004+0000 7f38074a5640 10 monclient: monmap: 2024-04-29T20:22:58.005 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:58.005 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.005 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:22:58.004858+0000 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:22:58.004858+0000 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:58.006 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.004+0000 7f38074a5640 1 Processor -- start 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.004+0000 7f38074a5640 1 -- start start 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.004+0000 7f38074a5640 10 monclient: init 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38000b5be0 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:58.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f38000b5fc0 0x7f3800146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): picked mon.noname-a con 0x7f38000b5fc0 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f38000aa460 0x7f38000b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): picked mon.noname-c con 0x7f38000aa460 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38000aa940 con 0x7f38000b5be0 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f3800146eb0 con 0x7f38000b5fc0 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f38000aabe0 con 0x7f38000aa460 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.005+0000 7f38074a5640 10 monclient(hunting): authenticate will time out at 1527.165001s 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3805a42640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f38000aa460 0x7f38000b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f38000b5fc0 0x7f3800146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f38000b5fc0 0x7f3800146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34386/0 (socket says 172.21.15.64:34386) 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 1 -- 172.21.15.64:0/3842755670 learned_addr learned my addr 172.21.15.64:0/3842755670 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:58.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 10 monclient(hunting): get_auth_request con 0x7f38000b5fc0 auth_method 0 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3804a40640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3805a42640 10 monclient(hunting): get_auth_request con 0x7f38000aa460 auth_method 0 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3805a42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3805a42640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.006+0000 7f3805a42640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.007+0000 7f3804a40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.007+0000 7f3804a40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.007+0000 7f3804a40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.007+0000 7f3805a42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.007+0000 7f3805a42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.007+0000 7f3805a42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2024-04-29T20:22:58.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 1 -- 172.21.15.64:0/3842755670 >> v2:172.21.15.167:3300/0 conn(0x7f38000aa460 msgr2=0x7f38000b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 1 --2- 172.21.15.64:0/3842755670 >> v2:172.21.15.167:3300/0 conn(0x7f38000aa460 0x7f38000b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 1 -- 172.21.15.64:0/3842755670 >> v1:172.21.15.167:6789/0 conn(0x7f38000b5be0 legacy=0x7f38000b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 10 monclient: _finish_hunting 0 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 1 monclient: found mon.noname-a 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.008+0000 7f3804a40640 1 -- 172.21.15.64:0/3842755670 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38000ab450 con 0x7f38000b5fc0 2024-04-29T20:22:58.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.011+0000 7f3804a40640 1 --2- 172.21.15.64:0/3842755670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38000b5fc0 0x7f3800146970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38000b4fa0 tx=0x7f37f002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee192d19b59be49 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:58.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 1 -- 172.21.15.64:0/3842755670 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0038070 con 0x7f38000b5fc0 2024-04-29T20:22:58.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:58.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: dump: 2024-04-29T20:22:58.017 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.017 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: _finish_auth 0 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: _check_auth_tickets 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 1 -- 172.21.15.64:0/3842755670 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f002f050 con 0x7f38000b5fc0 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:58.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 1 -- 172.21.15.64:0/3842755670 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0031970 con 0x7f38000b5fc0 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 5 monclient: authenticate success, global_id 4269 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 4 set_mon_vals no callback set 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f37f7fff640 10 monclient: dump: 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.020 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 10 monclient: shutdown 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 1 -- 172.21.15.64:0/3842755670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38000b5fc0 msgr2=0x7f3800146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 1 --2- 172.21.15.64:0/3842755670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38000b5fc0 0x7f3800146970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38000b4fa0 tx=0x7f37f002b7f0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 1 -- 172.21.15.64:0/3842755670 shutdown_connections 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 1 --2- 172.21.15.64:0/3842755670 >> v2:172.21.15.167:3300/0 conn(0x7f38000aa460 0x7f38000b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 1 --2- 172.21.15.64:0/3842755670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38000b5fc0 0x7f3800146970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.016+0000 7f38074a5640 1 -- 172.21.15.64:0/3842755670 >> 172.21.15.64:0/3842755670 conn(0x7f380006c940 msgr2=0x7f380006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:58.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.017+0000 7f38074a5640 1 -- 172.21.15.64:0/3842755670 shutdown_connections 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.017+0000 7f38074a5640 1 -- 172.21.15.64:0/3842755670 wait complete. 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.017+0000 7f38074a5640 10 monclient: build_initial_monmap 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.017+0000 7f38074a5640 10 monclient: monmap: 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.017+0000 7f38074a5640 1 librados: starting msgr at 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.017+0000 7f38074a5640 1 librados: starting objecter 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 Processor -- start 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 -- start start 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 librados: setting wanted keys 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 librados: calling monclient init 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient: init 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f38000aa460 0x7f3800144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): picked mon.noname-c con 0x7f38000aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 0x7f38001417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): picked mon.noname-b con 0x7f38000b5be0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3800141cf0 0x7f38000b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): picked mon.noname-a con 0x7f3800141cf0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f38000b27e0 con 0x7f3800141cf0 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f38000b2660 con 0x7f38000b5be0 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f380014dd00 con 0x7f38000aa460 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.018+0000 7f38074a5640 10 monclient(hunting): authenticate will time out at 1527.178336s 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 0x7f38001417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805a42640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3800141cf0 0x7f38000b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805241640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f38000aa460 0x7f3800144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 0x7f38001417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33388/0 (socket says 172.21.15.64:33388) 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 1 -- 172.21.15.64:0/3989817439 learned_addr learned my addr 172.21.15.64:0/3989817439 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:58.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 10 monclient(hunting): get_auth_request con 0x7f38000b5be0 auth_method 0 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3804a40640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805a42640 10 monclient(hunting): get_auth_request con 0x7f3800141cf0 auth_method 0 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805a42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805a42640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805a42640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805241640 10 monclient(hunting): get_auth_request con 0x7f38000aa460 auth_method 0 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805241640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805241640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.019+0000 7f3805241640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3804a40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3804a40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3804a40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3805a42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3805a42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3805a42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3805241640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3805241640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.020+0000 7f3805241640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-04-29T20:22:58.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 1 -- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f38000aa460 msgr2=0x7f3800144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f38000aa460 0x7f3800144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 1 -- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3800141cf0 msgr2=0x7f38000b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3800141cf0 0x7f38000b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 10 monclient: _finish_hunting 0 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 1 monclient: found mon.noname-b 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3804a40640 1 -- 172.21.15.64:0/3989817439 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3800142160 con 0x7f38000b5be0 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.023+0000 7f3805a42640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3800141cf0 0x7f38000b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.027+0000 7f3804a40640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 0x7f38001417b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f37f0000c00 tx=0x7f37f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.027+0000 7f37f5ffb640 10 client.?.objecter ms_handle_connect 0x7f38000b5be0 2024-04-29T20:22:58.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.027+0000 7f37f5ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:58.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0042070 con 0x7f38000b5be0 2024-04-29T20:22:58.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:58.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: dump: 2024-04-29T20:22:58.033 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.033 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: _finish_auth 0 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f003d660 con 0x7f38000b5be0 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:58.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0038040 con 0x7f38000b5be0 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f5ffb640 10 monclient: dump: 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.035 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f38074a5640 5 monclient: authenticate success, global_id 4285 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f38074a5640 10 monclient: _renew_subs 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f38074a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f37f77fe640 4 set_mon_vals no callback set 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.031+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38000acfa0 con 0x7f38000b5be0 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.032+0000 7f38074a5640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.032+0000 7f38074a5640 10 monclient: _renew_subs 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.032+0000 7f38074a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.032+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38000ad560 con 0x7f38000b5be0 2024-04-29T20:22:58.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.032+0000 7f38074a5640 1 librados: init done 2024-04-29T20:22:58.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.033+0000 7f38074a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:58.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.033+0000 7f38074a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:58.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.033+0000 7f38074a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.033+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f380004fce0 con 0x7f38000b5be0 2024-04-29T20:22:58.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.316+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f37f0045070 con 0x7f38000b5be0 2024-04-29T20:22:58.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.317+0000 7f37f5ffb640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f37e406e280 0x7f37e4070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.317+0000 7f3805241640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f37e406e280 0x7f37e4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.317+0000 7f3805241640 10 monclient: get_auth_request con 0x7f37e406e280 auth_method 0 2024-04-29T20:22:58.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.317+0000 7f37f5ffb640 10 client.4285.objecter ms_handle_connect 0x7f37e406e280 2024-04-29T20:22:58.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.317+0000 7f3805241640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f37e406e280 0x7f37e4070740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004f60 tx=0x7f37fc007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:58.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.322+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f37f0079bc0 con 0x7f38000b5be0 2024-04-29T20:22:58.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.322+0000 7f37f5ffb640 10 client.4285.objecter ms_dispatch 0x7f3800000b90 osd_map(25..25 src has 1..25) 2024-04-29T20:22:58.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.322+0000 7f37f5ffb640 3 client.4285.objecter handle_osd_map got epochs [25,25] > 0 2024-04-29T20:22:58.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.322+0000 7f37f5ffb640 3 client.4285.objecter handle_osd_map decoding full epoch 25 2024-04-29T20:22:58.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.322+0000 7f37f5ffb640 20 client.4285.objecter dump_active .. 0 homeless 2024-04-29T20:22:58.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.332+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f37f007bb20 con 0x7f38000b5be0 2024-04-29T20:22:58.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.332+0000 7f37f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:58.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.332+0000 7f37f5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:58.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.571+0000 7f38074a5640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-29T20:22:58.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.571+0000 7f38074a5640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-29T20:22:58.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.571+0000 7f38074a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.571+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f38000b8030 con 0x7f38000b5be0 2024-04-29T20:22:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.711+0000 7f37f5ffb640 1 -- 172.21.15.64:0/3989817439 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174144 (secure 0 0 0) 0x7f37f007fe20 con 0x7f38000b5be0 2024-04-29T20:22:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.711+0000 7f37f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-29T20:22:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.711+0000 7f37f5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:58.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:22:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 10 librados: watch_flush enter 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 10 librados: watch_flush exit 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 20 client.4285.objecter shutdown clearing up homeless session... 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 10 client.4285.objecter successfully canceled tick 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f37e406e280 msgr2=0x7f37e4070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f37e406e280 0x7f37e4070740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004f60 tx=0x7f37fc007e30 comp rx=0 tx=0).stop 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 10 monclient: shutdown 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 msgr2=0x7f38001417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 0x7f38001417b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f37f0000c00 tx=0x7f37f00046c0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 shutdown_connections 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f37e406e280 0x7f37e4070740 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3800141cf0 0x7f38000b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38000b5be0 0x7f38001417b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 --2- 172.21.15.64:0/3989817439 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f38000aa460 0x7f3800144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 >> 172.21.15.64:0/3989817439 conn(0x7f380006c940 msgr2=0x7f380013be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:58.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 shutdown_connections 2024-04-29T20:22:58.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.715+0000 7f38074a5640 1 -- 172.21.15.64:0/3989817439 wait complete. 2024-04-29T20:22:58.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.716+0000 7f38074a5640 1 librados: shutdown 2024-04-29T20:22:58.732 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":3900323440},{"type":"v1","addr":"172.21.15.167:6801","nonce":3900323440}]},"active_addr":"172.21.15.167:6801/3900323440","active_change":"2024-04-29T20:21:58.844913+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1512744575}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3452073622}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3311305940}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3807613187}]}]} 2024-04-29T20:22:58.738 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-29T20:22:58.738 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-29T20:22:58.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:58.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.848+0000 7efe87315640 10 monclient: get_monmap_and_config 2024-04-29T20:22:58.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.848+0000 7efe87315640 10 monclient: build_initial_monmap 2024-04-29T20:22:58.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.848+0000 7efe87315640 10 monclient: monmap: 2024-04-29T20:22:58.850 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:58.850 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:22:58.849658+0000 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:22:58.849658+0000 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.849+0000 7efe87315640 1 Processor -- start 2024-04-29T20:22:58.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.849+0000 7efe87315640 1 -- start start 2024-04-29T20:22:58.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.849+0000 7efe87315640 10 monclient: init 2024-04-29T20:22:58.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:58.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7efe800b5be0 0x7efe800b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): picked mon.noname-a con 0x7efe800b5be0 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe800b5fc0 addr v1:172.21.15.64:6789/0 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 0x7efe800b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): picked mon.noname-b con 0x7efe800aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe800aa940 con 0x7efe800b5fc0 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7efe800b3420 con 0x7efe800b5be0 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7efe800aabe0 con 0x7efe800aa460 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:58.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.850+0000 7efe87315640 10 monclient(hunting): authenticate will time out at 1528.009969s 2024-04-29T20:22:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7efe800b5be0 0x7efe800b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7efe800b5be0 0x7efe800b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34406/0 (socket says 172.21.15.64:34406) 2024-04-29T20:22:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 1 -- 172.21.15.64:0/3384104348 learned_addr learned my addr 172.21.15.64:0/3384104348 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 10 monclient(hunting): get_auth_request con 0x7efe800b5be0 auth_method 0 2024-04-29T20:22:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.859+0000 7efe850b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.865+0000 7efe850b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.865+0000 7efe850b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.865+0000 7efe850b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 1 -- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 msgr2=0x7efe800b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 1 --2- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 0x7efe800b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 1 -- 172.21.15.64:0/3384104348 >> v1:172.21.15.64:6789/0 conn(0x7efe800b5fc0 legacy=0x7efe80146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 10 monclient: _finish_hunting 0 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 1 monclient: found mon.noname-a 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.867+0000 7efe850b1640 1 -- 172.21.15.64:0/3384104348 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe800ab450 con 0x7efe800b5be0 2024-04-29T20:22:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.871+0000 7efe850b1640 1 --2- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe800b2ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009870 tx=0x7efe7c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b35e235b055c5a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 1 -- 172.21.15.64:0/3384104348 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0317c0 con 0x7efe800b5be0 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: dump: 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: _finish_auth 0 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: _check_auth_tickets 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 1 -- 172.21.15.64:0/3384104348 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7c037070 con 0x7efe800b5be0 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 1 -- 172.21.15.64:0/3384104348 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c031b80 con 0x7efe800b5be0 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe77fff640 10 monclient: dump: 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe87315640 5 monclient: authenticate success, global_id 4275 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe87315640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.875+0000 7efe87315640 4 set_mon_vals no callback set 2024-04-29T20:22:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 10 monclient: shutdown 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 -- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 msgr2=0x7efe800b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 --2- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe800b2ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009870 tx=0x7efe7c02b860 comp rx=0 tx=0).stop 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 -- 172.21.15.64:0/3384104348 shutdown_connections 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 --2- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 0x7efe800b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 --2- 172.21.15.64:0/3384104348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe800b2ee0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 -- 172.21.15.64:0/3384104348 >> 172.21.15.64:0/3384104348 conn(0x7efe8006c940 msgr2=0x7efe8006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 -- 172.21.15.64:0/3384104348 shutdown_connections 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.876+0000 7efe87315640 1 -- 172.21.15.64:0/3384104348 wait complete. 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.877+0000 7efe87315640 10 monclient: build_initial_monmap 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.877+0000 7efe87315640 10 monclient: monmap: 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:22:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:58.884 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:58.884 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:58.884 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.877+0000 7efe87315640 1 librados: starting msgr at 2024-04-29T20:22:58.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.877+0000 7efe87315640 1 librados: starting objecter 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 Processor -- start 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 -- start start 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 librados: setting wanted keys 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 librados: calling monclient init 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient: init 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 0x7efe801444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): picked mon.noname-c con 0x7efe800aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe80143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): picked mon.noname-a con 0x7efe800b5be0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 0x7efe801379e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): picked mon.noname-b con 0x7efe800ad630 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efe800b27e0 con 0x7efe800b5be0 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efe800b2660 con 0x7efe800ad630 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7efe8014dd00 con 0x7efe800aa460 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.878+0000 7efe87315640 10 monclient(hunting): authenticate will time out at 1528.038351s 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.879+0000 7efe858b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 0x7efe801379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.880+0000 7efe858b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 0x7efe801379e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33402/0 (socket says 172.21.15.64:33402) 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.880+0000 7efe858b2640 1 -- 172.21.15.64:0/4081280450 learned_addr learned my addr 172.21.15.64:0/4081280450 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.880+0000 7efe858b2640 10 monclient(hunting): get_auth_request con 0x7efe800ad630 auth_method 0 2024-04-29T20:22:58.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.880+0000 7efe858b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:58.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.880+0000 7efe858b2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:58.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.880+0000 7efe858b2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.881+0000 7efe858b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.881+0000 7efe858b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.881+0000 7efe858b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.881+0000 7efe848b0640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe80143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 1 -- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 msgr2=0x7efe801444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 0x7efe801444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 1 -- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 msgr2=0x7efe80143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:58.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe80143b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 10 monclient: _finish_hunting 0 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 1 monclient: found mon.noname-b 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.884+0000 7efe858b2640 1 -- 172.21.15.64:0/4081280450 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe80145060 con 0x7efe800ad630 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.887+0000 7efe858b2640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 0x7efe801379e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efe7800b9e0 tx=0x7efe7800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.887+0000 7efe75ffb640 10 client.?.objecter ms_handle_connect 0x7efe800ad630 2024-04-29T20:22:58.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.887+0000 7efe75ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78019070 con 0x7efe800ad630 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: dump: 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.892 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: _finish_auth 0 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7800cac0 con 0x7efe800ad630 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78014440 con 0x7efe800ad630 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 5 monclient: authenticate success, global_id 4288 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 10 monclient: _renew_subs 2024-04-29T20:22:58.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe800acbe0 con 0x7efe800ad630 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe777fe640 4 set_mon_vals no callback set 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe75ffb640 10 monclient: dump: 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:58.894 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:58.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:58.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 10 monclient: _renew_subs 2024-04-29T20:22:58.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe80144b40 con 0x7efe800ad630 2024-04-29T20:22:58.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.890+0000 7efe87315640 1 librados: init done 2024-04-29T20:22:58.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.892+0000 7efe87315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:58.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.892+0000 7efe87315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:58.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.892+0000 7efe87315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:58.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.892+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe80144d40 con 0x7efe800ad630 2024-04-29T20:22:58.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.909+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe7801c070 con 0x7efe800ad630 2024-04-29T20:22:58.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.910+0000 7efe75ffb640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7efe6406e280 0x7efe64070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:58.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.910+0000 7efe850b1640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7efe6406e280 0x7efe64070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:58.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.910+0000 7efe850b1640 10 monclient: get_auth_request con 0x7efe6406e280 auth_method 0 2024-04-29T20:22:58.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.911+0000 7efe850b1640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7efe6406e280 0x7efe64070740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efe7c036000 tx=0x7efe7c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:58.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.911+0000 7efe75ffb640 10 client.4288.objecter ms_handle_connect 0x7efe6406e280 2024-04-29T20:22:58.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.913+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7efe78057b10 con 0x7efe800ad630 2024-04-29T20:22:58.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.913+0000 7efe75ffb640 10 client.4288.objecter ms_dispatch 0x7efe80000b90 osd_map(25..25 src has 1..25) 2024-04-29T20:22:58.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.913+0000 7efe75ffb640 3 client.4288.objecter handle_osd_map got epochs [25,25] > 0 2024-04-29T20:22:58.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.913+0000 7efe75ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 25 2024-04-29T20:22:58.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:58.913+0000 7efe75ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-04-29T20:22:59.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.171+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe80144d40 con 0x7efe800ad630 2024-04-29T20:22:59.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.171+0000 7efe75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:59.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.171+0000 7efe75ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:22:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.363+0000 7efe87315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.363+0000 7efe87315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.363+0000 7efe87315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.363+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe80071f90 con 0x7efe800ad630 2024-04-29T20:22:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.397+0000 7efe75ffb640 1 -- 172.21.15.64:0/4081280450 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+14742 (secure 0 0 0) 0x7efe78023020 con 0x7efe800ad630 2024-04-29T20:22:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.397+0000 7efe75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:22:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.397+0000 7efe75ffb640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:22:59.399 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:22:59.399 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":26,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:22:58.769885+0000","last_up_change":"2024-04-29T20:22:54.303806+0000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T20:22:41.558205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-29T20:22:56.229418+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6810","nonce":2796311601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6815","nonce":2796311601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6823","nonce":2796311601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6821","nonce":2796311601}]},"public_addr":"172.21.15.64:6810/2796311601","cluster_addr":"172.21.15.64:6815/2796311601","heartbeat_back_addr":"172.21.15.64:6823/2796311601","heartbeat_front_addr":"172.21.15.64:6821/2796311601","state":["exists","up"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6811","nonce":3421706551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6814","nonce":3421706551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6820","nonce":3421706551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6817","nonce":3421706551}]},"public_addr":"172.21.15.64:6811/3421706551","cluster_addr":"172.21.15.64:6814/3421706551","heartbeat_back_addr":"172.21.15.64:6820/3421706551","heartbeat_front_addr":"172.21.15.64:6817/3421706551","state":["exists","up"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6801","nonce":2373308255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6803","nonce":2373308255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6807","nonce":2373308255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6805","nonce":2373308255}]},"public_addr":"172.21.15.64:6801/2373308255","cluster_addr":"172.21.15.64:6803/2373308255","heartbeat_back_addr":"172.21.15.64:6807/2373308255","heartbeat_front_addr":"172.21.15.64:6805/2373308255","state":["exists","up"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6825","nonce":471606907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6827","nonce":471606907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6831","nonce":471606907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6829","nonce":471606907}]},"public_addr":"172.21.15.64:6825/471606907","cluster_addr":"172.21.15.64:6827/471606907","heartbeat_back_addr":"172.21.15.64:6831/471606907","heartbeat_front_addr":"172.21.15.64:6829/471606907","state":["exists","up"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6827","nonce":3754201859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6829","nonce":3754201859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6833","nonce":3754201859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6831","nonce":3754201859}]},"public_addr":"172.21.15.167:6827/3754201859","cluster_addr":"172.21.15.167:6829/3754201859","heartbeat_back_addr":"172.21.15.167:6833/3754201859","heartbeat_front_addr":"172.21.15.167:6831/3754201859","state":["exists","up"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6803","nonce":3447508053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6805","nonce":3447508053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6809","nonce":3447508053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6807","nonce":3447508053}]},"public_addr":"172.21.15.167:6803/3447508053","cluster_addr":"172.21.15.167:6805/3447508053","heartbeat_back_addr":"172.21.15.167:6809/3447508053","heartbeat_front_addr":"172.21.15.167:6807/3447508053","state":["exists","up"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6819","nonce":3916781155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6821","nonce":3916781155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6825","nonce":3916781155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6823","nonce":3916781155}]},"public_addr":"172.21.15.167:6819/3916781155","cluster_addr":"172.21.15.167:6821/3916781155","heartbeat_back_addr":"172.21.15.167:6825/3916781155","heartbeat_front_addr":"172.21.15.167:6823/3916781155","state":["exists","up"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6811","nonce":2258625350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6813","nonce":2258625350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6817","nonce":2258625350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6815","nonce":2258625350}]},"public_addr":"172.21.15.167:6811/2258625350","cluster_addr":"172.21.15.167:6813/2258625350","heartbeat_back_addr":"172.21.15.167:6817/2258625350","heartbeat_front_addr":"172.21.15.167:6815/2258625350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:40.154276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.540513+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:38.282652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.442559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.557528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.720225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:35.815755+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.803337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:22:59.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 10 librados: watch_flush enter 2024-04-29T20:22:59.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 10 librados: watch_flush exit 2024-04-29T20:22:59.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 20 client.4288.objecter shutdown clearing up homeless session... 2024-04-29T20:22:59.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 10 client.4288.objecter successfully canceled tick 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7efe6406e280 msgr2=0x7efe64070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7efe6406e280 0x7efe64070740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efe7c036000 tx=0x7efe7c030000 comp rx=0 tx=0).stop 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 10 monclient: shutdown 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 msgr2=0x7efe801379e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 0x7efe801379e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efe7800b9e0 tx=0x7efe7800bea0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 shutdown_connections 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7efe6406e280 0x7efe64070740 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe800ad630 0x7efe801379e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe800b5be0 0x7efe80143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 --2- 172.21.15.64:0/4081280450 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7efe800aa460 0x7efe801444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 >> 172.21.15.64:0/4081280450 conn(0x7efe8006c940 msgr2=0x7efe8006cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:59.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 shutdown_connections 2024-04-29T20:22:59.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 -- 172.21.15.64:0/4081280450 wait complete. 2024-04-29T20:22:59.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.399+0000 7efe87315640 1 librados: shutdown 2024-04-29T20:22:59.412 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-29T20:22:59.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-29T20:22:59.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.513+0000 7fc6a6585640 10 monclient: get_monmap_and_config 2024-04-29T20:22:59.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.513+0000 7fc6a6585640 10 monclient: build_initial_monmap 2024-04-29T20:22:59.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.513+0000 7fc6a6585640 10 monclient: monmap: 2024-04-29T20:22:59.514 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:22:59.514101+0000 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:22:59.514101+0000 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:22:59.515 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:22:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:22:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:22:59.518 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.513+0000 7fc6a6585640 1 Processor -- start 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 1 -- start start 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient: init 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6a00b5be0 addr v1:172.21.15.167:6789/0 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc6a00b5fc0 0x7fc6a0146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a00b5fc0 addr v2:172.21.15.167:3300/0 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc6a00aa460 0x7fc6a00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): picked mon.noname-a con 0x7fc6a00aa460 addr v2:172.21.15.64:3300/0 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6a00aa940 con 0x7fc6a00b5be0 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a0146eb0 con 0x7fc6a00aa460 2024-04-29T20:22:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a00aabe0 con 0x7fc6a00b5fc0 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.514+0000 7fc6a6585640 10 monclient(hunting): authenticate will time out at 1528.674233s 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc6a00aa460 0x7fc6a00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc69f7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc6a00b5fc0 0x7fc6a0146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc6a00aa460 0x7fc6a00b2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34422/0 (socket says 172.21.15.64:34422) 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 1 -- 172.21.15.64:0/2895992354 learned_addr learned my addr 172.21.15.64:0/2895992354 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 10 monclient(hunting): get_auth_request con 0x7fc6a00aa460 auth_method 0 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc6a4b22640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc69f7fe640 10 monclient(hunting): get_auth_request con 0x7fc6a00b5fc0 auth_method 0 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc69f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.515+0000 7fc69f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.516+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.516+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.516+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.516+0000 7fc69f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.516+0000 7fc69f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.516+0000 7fc69f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.517+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.517+0000 7fc6a4b22640 1 -- 172.21.15.64:0/2895992354 >> v2:172.21.15.167:3300/0 conn(0x7fc6a00b5fc0 msgr2=0x7fc6a0146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.517+0000 7fc6a4b22640 1 --2- 172.21.15.64:0/2895992354 >> v2:172.21.15.167:3300/0 conn(0x7fc6a00b5fc0 0x7fc6a0146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.517+0000 7fc6a4b22640 1 -- 172.21.15.64:0/2895992354 >> v1:172.21.15.167:6789/0 conn(0x7fc6a00b5be0 legacy=0x7fc6a00b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.518+0000 7fc6a4b22640 10 monclient: _finish_hunting 0 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.518+0000 7fc6a4b22640 1 monclient: found mon.noname-a 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.518+0000 7fc6a4b22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.518+0000 7fc6a4b22640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:22:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.518+0000 7fc6a4b22640 1 -- 172.21.15.64:0/2895992354 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a00ab450 con 0x7fc6a00aa460 2024-04-29T20:22:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.520+0000 7fc6a4b22640 1 --2- 172.21.15.64:0/2895992354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00aa460 0x7fc6a00b2120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00abeb0 tx=0x7fc69802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48903b40c839887 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 1 -- 172.21.15.64:0/2895992354 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69802c9f0 con 0x7fc6a00aa460 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: dump: 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: _finish_auth 0 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: _check_auth_tickets 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 1 -- 172.21.15.64:0/2895992354 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69802cb90 con 0x7fc6a00aa460 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 1 -- 172.21.15.64:0/2895992354 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6980343f0 con 0x7fc6a00aa460 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc6a6585640 5 monclient: authenticate success, global_id 4281 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc6a6585640 10 monclient: get_monmap_and_config success 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc6a6585640 4 set_mon_vals no callback set 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.523+0000 7fc69effd640 10 monclient: dump: 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:59.527 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 10 monclient: shutdown 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 -- 172.21.15.64:0/2895992354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00aa460 msgr2=0x7fc6a00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 --2- 172.21.15.64:0/2895992354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00aa460 0x7fc6a00b2120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00abeb0 tx=0x7fc69802be50 comp rx=0 tx=0).stop 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 -- 172.21.15.64:0/2895992354 shutdown_connections 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 --2- 172.21.15.64:0/2895992354 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00aa460 0x7fc6a00b2120 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 --2- 172.21.15.64:0/2895992354 >> v2:172.21.15.167:3300/0 conn(0x7fc6a00b5fc0 0x7fc6a0146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 -- 172.21.15.64:0/2895992354 >> 172.21.15.64:0/2895992354 conn(0x7fc6a006c940 msgr2=0x7fc6a006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 -- 172.21.15.64:0/2895992354 shutdown_connections 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.524+0000 7fc6a6585640 1 -- 172.21.15.64:0/2895992354 wait complete. 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.525+0000 7fc6a6585640 10 monclient: build_initial_monmap 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.525+0000 7fc6a6585640 10 monclient: monmap: 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:22:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.525+0000 7fc6a6585640 1 librados: starting msgr at 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.525+0000 7fc6a6585640 1 librados: starting objecter 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.525+0000 7fc6a6585640 1 Processor -- start 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 -- start start 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 librados: setting wanted keys 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 librados: calling monclient init 2024-04-29T20:22:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient: init 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient: _reopen_session rank -1 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6a00aa460 0x7fc6a01447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): picked mon.noname-b con 0x7fc6a00aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 0x7fc6a0144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): picked mon.noname-a con 0x7fc6a00b5be0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc6a0141270 0x7fc6a01416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a0141270 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a00b27e0 con 0x7fc6a00b5be0 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a00b2660 con 0x7fc6a00aa460 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): start opening mon connection 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a014dd00 con 0x7fc6a0141270 2024-04-29T20:22:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): _renew_subs 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.526+0000 7fc6a6585640 10 monclient(hunting): authenticate will time out at 1528.686173s 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 0x7fc6a0144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc6a4b22640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc6a0141270 0x7fc6a01416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6a00aa460 0x7fc6a01447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 0x7fc6a0144000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34428/0 (socket says 172.21.15.64:34428) 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 1 -- 172.21.15.64:0/921883535 learned_addr learned my addr 172.21.15.64:0/921883535 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 10 monclient(hunting): get_auth_request con 0x7fc6a00b5be0 auth_method 0 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc6a4b22640 10 monclient(hunting): get_auth_request con 0x7fc6a0141270 auth_method 0 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc6a4b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc6a4b22640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc6a4b22640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69ffff640 10 monclient(hunting): get_auth_request con 0x7fc6a00aa460 auth_method 0 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.527+0000 7fc69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc69f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc69f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc69f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc69ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc69ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc69ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:22:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.528+0000 7fc6a4b22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:22:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.530+0000 7fc69ffff640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2024-04-29T20:22:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 1 -- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc6a0141270 msgr2=0x7fc6a01416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc6a0141270 0x7fc6a01416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 1 -- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 msgr2=0x7fc6a0144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 0x7fc6a0144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 10 monclient: _finish_hunting 0 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 1 monclient: found mon.noname-b 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69ffff640 1 -- 172.21.15.64:0/921883535 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a0145370 con 0x7fc6a00aa460 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.531+0000 7fc69f7fe640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 0x7fc6a0144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.533+0000 7fc69ffff640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6a00aa460 0x7fc6a01447c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c370 tx=0x7fc69000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.533+0000 7fc69cff9640 10 client.?.objecter ms_handle_connect 0x7fc6a00aa460 2024-04-29T20:22:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.533+0000 7fc69cff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:22:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6900052d0 con 0x7fc6a00aa460 2024-04-29T20:22:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:22:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: dump: 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: _finish_auth 0 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: _check_auth_tickets 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc690005490 con 0x7fc6a00aa460 2024-04-29T20:22:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690007260 con 0x7fc6a00aa460 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.536+0000 7fc69cff9640 10 monclient: dump: 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:22:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 5 monclient: authenticate success, global_id 4297 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 monclient: _renew_subs 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a00ad290 con 0x7fc6a00aa460 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc69e7fc640 4 set_mon_vals no callback set 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 monclient: _renew_subs 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a0141be0 con 0x7fc6a00aa460 2024-04-29T20:22:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 1 librados: init done 2024-04-29T20:22:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:22:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.537+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a004fce0 con 0x7fc6a00aa460 2024-04-29T20:22:59.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.557+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc690007400 con 0x7fc6a00aa460 2024-04-29T20:22:59.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.557+0000 7fc69cff9640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc68c06e2d0 0x7fc68c070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:22:59.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.557+0000 7fc69f7fe640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc68c06e2d0 0x7fc68c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:22:59.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.557+0000 7fc69f7fe640 10 monclient: get_auth_request con 0x7fc68c06e2d0 auth_method 0 2024-04-29T20:22:59.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.558+0000 7fc69cff9640 10 client.4297.objecter ms_handle_connect 0x7fc68c06e2d0 2024-04-29T20:22:59.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.558+0000 7fc69f7fe640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc68c06e2d0 0x7fc68c070790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00b1a80 tx=0x7fc694009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:22:59.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.795+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5210+0+0 (secure 0 0 0) 0x7fc690057290 con 0x7fc6a00aa460 2024-04-29T20:22:59.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.795+0000 7fc69cff9640 10 client.4297.objecter ms_dispatch 0x7fc6a0000b90 osd_map(26..26 src has 1..26) 2024-04-29T20:22:59.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.795+0000 7fc69cff9640 3 client.4297.objecter handle_osd_map got epochs [26,26] > 0 2024-04-29T20:22:59.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.795+0000 7fc69cff9640 3 client.4297.objecter handle_osd_map decoding full epoch 26 2024-04-29T20:22:59.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.795+0000 7fc69cff9640 20 client.4297.objecter dump_active .. 0 homeless 2024-04-29T20:22:59.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.805+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6a00ad290 con 0x7fc6a00aa460 2024-04-29T20:22:59.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.805+0000 7fc69cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:22:59.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:22:59.805+0000 7fc69cff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:00.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.032+0000 7fc6a6585640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:23:00.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.032+0000 7fc6a6585640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:23:00.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.032+0000 7fc6a6585640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.032+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6a00b84d0 con 0x7fc6a00aa460 2024-04-29T20:23:00.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.178+0000 7fc69cff9640 1 -- 172.21.15.64:0/921883535 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+14729 (secure 0 0 0) 0x7fc6900591f0 con 0x7fc6a00aa460 2024-04-29T20:23:00.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.178+0000 7fc69cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:23:00.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.178+0000 7fc69cff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:23:00.181 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:22:59.922795+0000","last_up_change":"2024-04-29T20:22:54.303806+0000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T20:22:41.558205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-29T20:22:56.229418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6810","nonce":2796311601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6815","nonce":2796311601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6823","nonce":2796311601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6821","nonce":2796311601}]},"public_addr":"172.21.15.64:6810/2796311601","cluster_addr":"172.21.15.64:6815/2796311601","heartbeat_back_addr":"172.21.15.64:6823/2796311601","heartbeat_front_addr":"172.21.15.64:6821/2796311601","state":["exists","up"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6811","nonce":3421706551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6814","nonce":3421706551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6820","nonce":3421706551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6817","nonce":3421706551}]},"public_addr":"172.21.15.64:6811/3421706551","cluster_addr":"172.21.15.64:6814/3421706551","heartbeat_back_addr":"172.21.15.64:6820/3421706551","heartbeat_front_addr":"172.21.15.64:6817/3421706551","state":["exists","up"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6801","nonce":2373308255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6803","nonce":2373308255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6807","nonce":2373308255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6805","nonce":2373308255}]},"public_addr":"172.21.15.64:6801/2373308255","cluster_addr":"172.21.15.64:6803/2373308255","heartbeat_back_addr":"172.21.15.64:6807/2373308255","heartbeat_front_addr":"172.21.15.64:6805/2373308255","state":["exists","up"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6825","nonce":471606907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6827","nonce":471606907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6831","nonce":471606907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6829","nonce":471606907}]},"public_addr":"172.21.15.64:6825/471606907","cluster_addr":"172.21.15.64:6827/471606907","heartbeat_back_addr":"172.21.15.64:6831/471606907","heartbeat_front_addr":"172.21.15.64:6829/471606907","state":["exists","up"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6827","nonce":3754201859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6829","nonce":3754201859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6833","nonce":3754201859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6831","nonce":3754201859}]},"public_addr":"172.21.15.167:6827/3754201859","cluster_addr":"172.21.15.167:6829/3754201859","heartbeat_back_addr":"172.21.15.167:6833/3754201859","heartbeat_front_addr":"172.21.15.167:6831/3754201859","state":["exists","up"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6803","nonce":3447508053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6805","nonce":3447508053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6809","nonce":3447508053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6807","nonce":3447508053}]},"public_addr":"172.21.15.167:6803/3447508053","cluster_addr":"172.21.15.167:6805/3447508053","heartbeat_back_addr":"172.21.15.167:6809/3447508053","heartbeat_front_addr":"172.21.15.167:6807/3447508053","state":["exists","up"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6819","nonce":3916781155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6821","nonce":3916781155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6825","nonce":3916781155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6823","nonce":3916781155}]},"public_addr":"172.21.15.167:6819/3916781155","cluster_addr":"172.21.15.167:6821/3916781155","heartbeat_back_addr":"172.21.15.167:6825/3916781155","heartbeat_front_addr":"172.21.15.167:6823/3916781155","state":["exists","up"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6811","nonce":2258625350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6813","nonce":2258625350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6817","nonce":2258625350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6815","nonce":2258625350}]},"public_addr":"172.21.15.167:6811/2258625350","cluster_addr":"172.21.15.167:6813/2258625350","heartbeat_back_addr":"172.21.15.167:6817/2258625350","heartbeat_front_addr":"172.21.15.167:6815/2258625350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:40.154276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.540513+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:38.282652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.442559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.557528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.720225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:35.815755+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.803337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:23:00.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.180+0000 7fc6a6585640 10 librados: watch_flush enter 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.180+0000 7fc6a6585640 10 librados: watch_flush exit 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.180+0000 7fc6a6585640 20 client.4297.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.180+0000 7fc6a6585640 10 client.4297.objecter successfully canceled tick 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc68c06e2d0 msgr2=0x7fc68c070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc68c06e2d0 0x7fc68c070790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00b1a80 tx=0x7fc694009190 comp rx=0 tx=0).stop 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 10 monclient: shutdown 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6a00aa460 msgr2=0x7fc6a01447c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6a00aa460 0x7fc6a01447c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c370 tx=0x7fc69000c830 comp rx=0 tx=0).stop 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 shutdown_connections 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc68c06e2d0 0x7fc68c070790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc6a0141270 0x7fc6a01416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a00b5be0 0x7fc6a0144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 --2- 172.21.15.64:0/921883535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6a00aa460 0x7fc6a01447c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 >> 172.21.15.64:0/921883535 conn(0x7fc6a006c940 msgr2=0x7fc6a013c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 shutdown_connections 2024-04-29T20:23:00.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 -- 172.21.15.64:0/921883535 wait complete. 2024-04-29T20:23:00.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.181+0000 7fc6a6585640 1 librados: shutdown 2024-04-29T20:23:00.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-29T20:23:00.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-29T20:23:00.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-29T20:23:00.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-29T20:23:00.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-29T20:23:00.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-29T20:23:00.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-29T20:23:00.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.372+0000 7f81218e4640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.372+0000 7f81218e4640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.372+0000 7f81218e4640 10 monclient: monmap: 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.373461+0000 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.373461+0000 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.374 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.373+0000 7fb7ef957640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.373+0000 7fb7ef957640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.373+0000 7fb7ef957640 10 monclient: monmap: 2024-04-29T20:23:00.375 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.376 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.376 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.374282+0000 2024-04-29T20:23:00.376 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.374282+0000 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.377 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.373+0000 7f81218e4640 1 Processor -- start 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.373+0000 7f81218e4640 1 -- start start 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.373+0000 7f81218e4640 10 monclient: init 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f811c0b2ee0 0x7f811c144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): picked mon.noname-c con 0x7f811c0b2ee0 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f811c0b37b0 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f811c150200 0x7f811c1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): picked mon.noname-a con 0x7f811c150200 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f811c06ef20 con 0x7f811c0b37b0 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f811c0b32c0 con 0x7f811c150200 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f811c06ecd0 con 0x7f811c0b2ee0 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7f81218e4640 10 monclient(hunting): authenticate will time out at 1529.534444s 2024-04-29T20:23:00.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7fb7ef957640 1 Processor -- start 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7fb7ef957640 1 -- start start 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7fb7ef957640 10 monclient: init 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7febf1406640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7febf1406640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.374+0000 7febf1406640 10 monclient: monmap: 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.375543+0000 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.375543+0000 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.380 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7febf1406640 1 Processor -- start 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7febf1406640 1 -- start start 2024-04-29T20:23:00.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7febf1406640 10 monclient: init 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8144a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e80aadd0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7e80abe50 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7e80ab1b0 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80ad020 con 0x7fb7e80ab1b0 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80ad290 con 0x7fb7e80abe50 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e80ad6b0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.375+0000 7fb7ef957640 10 monclient(hunting): authenticate will time out at 1529.535378s 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7fb7ecef2640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb7e80abe50 0x7fb7e8144f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40580/0 (socket says 172.21.15.64:40580) 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7fb7ecef2640 1 -- 172.21.15.64:0/2629760342 learned_addr learned my addr 172.21.15.64:0/2629760342 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7f811a7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f811c0b37b0 0x7f811c0b3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51820/0 (socket says 172.21.15.64:51820) 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7f811a7fc640 1 -- 172.21.15.64:0/1188108462 learned_addr learned my addr 172.21.15.64:0/1188108462 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7febec073130 0x7febec0aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): picked mon.noname-c con 0x7febec073130 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febec157b60 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec0afc70 0x7febec15f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): picked mon.noname-b con 0x7febec0afc70 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febec0ac060 con 0x7febec157b60 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7febec073510 con 0x7febec073130 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7febec0aaac0 con 0x7febec0afc70 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.376+0000 7febf1406640 10 monclient(hunting): authenticate will time out at 1529.536255s 2024-04-29T20:23:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.377+0000 7f811affd640 1 --2- 172.21.15.64:0/1188108462 >> v2:172.21.15.167:3300/0 conn(0x7f811c0b2ee0 0x7f811c144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f811b7fe640 1 --2- 172.21.15.64:0/1188108462 >> v2:172.21.15.64:3300/0 conn(0x7f811c150200 0x7f811c1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f811b7fe640 10 monclient(hunting): get_auth_request con 0x7f811c150200 auth_method 0 2024-04-29T20:23:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f811b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f811b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f811b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7fb7ed6f3640 1 --2- 172.21.15.64:0/2629760342 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7fb7ed6f3640 10 monclient(hunting): get_auth_request con 0x7fb7e80aadd0 auth_method 0 2024-04-29T20:23:00.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f947fa09640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7fb7ed6f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7fb7ed6f3640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7fb7ed6f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f947fa09640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f947fa09640 10 monclient: monmap: 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.378944+0000 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.378944+0000 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.388 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.379+0000 7f947fa09640 1 Processor -- start 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.379+0000 7febea7fc640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7febec157b60 0x7febec0af560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40588/0 (socket says 172.21.15.64:40588) 2024-04-29T20:23:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.379+0000 7febea7fc640 1 -- 172.21.15.64:0/840354360 learned_addr learned my addr 172.21.15.64:0/840354360 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.379+0000 7f947fa09640 1 -- start start 2024-04-29T20:23:00.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.379+0000 7f947fa09640 10 monclient: init 2024-04-29T20:23:00.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeaffd640 1 --2- 172.21.15.64:0/840354360 >> v2:172.21.15.167:3300/0 conn(0x7febec073130 0x7febec0aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7dffff640 1 -- 172.21.15.64:0/2629760342 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256480074 0 0) 0x7fb7e4001110 con 0x7fb7e80abe50 2024-04-29T20:23:00.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7dffff640 10 monclient(hunting): _negotiate 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7dffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7dffff640 10 monclient(hunting): my global_id is 4300 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7dffff640 1 -- 172.21.15.64:0/2629760342 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7d4003520 con 0x7fb7e80abe50 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f811affd640 10 monclient(hunting): get_auth_request con 0x7f811c0b2ee0 auth_method 0 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f811affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f811affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f811affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeb7fe640 1 --2- 172.21.15.64:0/840354360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec0afc70 0x7febec15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeb7fe640 10 monclient(hunting): get_auth_request con 0x7febec0afc70 auth_method 0 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeaffd640 10 monclient(hunting): get_auth_request con 0x7febec073130 auth_method 0 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeaffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7febeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 0x7f947813c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient(hunting): picked mon.noname-b con 0x7f94780ab1a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94780abc10 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f947813c910 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94780ad110 con 0x7f94780abc10 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94780ad3f0 con 0x7f947813c910 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f94780ab580 con 0x7f94780ab1a0 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f947fa09640 10 monclient(hunting): authenticate will time out at 1529.540524s 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f373d6e8640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f373d6e8640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f373d6e8640 10 monclient: monmap: 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.381410+0000 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.381410+0000 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.394 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f3f00af3640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f3f00af3640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.378+0000 7f3f00af3640 10 monclient: monmap: 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.379478+0000 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.379478+0000 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.396 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f3f00af3640 1 Processor -- start 2024-04-29T20:23:00.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7febeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7febeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7febeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f811b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f811b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f811b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f28f7fff640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f28f7fff640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f373d6e8640 1 Processor -- start 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.380+0000 7f28f7fff640 10 monclient: monmap: 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.381371+0000 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.381371+0000 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.399 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f373d6e8640 1 -- start start 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f373d6e8640 10 monclient: init 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7febeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7febeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7febeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f28f7fff640 1 Processor -- start 2024-04-29T20:23:00.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f28f7fff640 1 -- start start 2024-04-29T20:23:00.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f28f7fff640 10 monclient: init 2024-04-29T20:23:00.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-29T20:23:00.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f37300c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): picked mon.noname-b con 0x7f3730006520 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3730032c60 0x7f37300c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): picked mon.noname-a con 0x7f3730032c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f3f00af3640 1 -- start start 2024-04-29T20:23:00.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.381+0000 7f3f00af3640 10 monclient: init 2024-04-29T20:23:00.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f37300cb270 0x7f37300cb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): picked mon.noname-c con 0x7f37300cb270 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc0ab1a0 0x7f3efc13c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): picked mon.noname-b con 0x7f3efc0ab1a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3efc0abc10 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f3730006900 con 0x7f3730032c60 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f3730033cc0 con 0x7f37300cb270 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f3730032810 con 0x7f3730006520 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f373d6e8640 10 monclient(hunting): authenticate will time out at 1529.542259s 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f3efc13c910 0x7f3efc0ae390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): picked mon.noname-c con 0x7f3efc13c910 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3efc0ad110 con 0x7f3efc0abc10 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f3efc0ab580 con 0x7f3efc13c910 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc0ac160 con 0x7f3efc0ab1a0 2024-04-29T20:23:00.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.382+0000 7f3f00af3640 10 monclient(hunting): authenticate will time out at 1529.542532s 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f28f80aab10 0x7f28f80aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f28f80aab10 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f28f80ad460 0x7f28f814b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f28f80ad460 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f814b890 0x7f28f81477e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f28f814b890 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f28f80aca80 con 0x7f28f80ad460 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f28f80acd60 con 0x7f28f80aab10 2024-04-29T20:23:00.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f28f80ad040 con 0x7f28f814b890 2024-04-29T20:23:00.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.383+0000 7f28f7fff640 10 monclient(hunting): authenticate will time out at 1529.542926s 2024-04-29T20:23:00.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-04-29T20:23:00.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 1 -- 172.21.15.64:0/2629760342 >> v1:172.21.15.167:6789/0 conn(0x7fb7e80abe50 legacy=0x7fb7e8144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 1 -- 172.21.15.64:0/2629760342 >> v1:172.21.15.64:6789/0 conn(0x7fb7e80ab1b0 legacy=0x7fb7e814b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7fb7ed6f3640 1 -- 172.21.15.64:0/2629760342 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e814b960 con 0x7fb7e80aadd0 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 0x7f947813c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 0x7f947813c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36624/0 (socket says 172.21.15.64:36624) 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 1 -- 172.21.15.64:0/1125711392 learned_addr learned my addr 172.21.15.64:0/1125711392 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 10 monclient(hunting): get_auth_request con 0x7f94780ab1a0 auth_method 0 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.384+0000 7f947d7a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 1 -- 172.21.15.64:0/1188108462 >> v2:172.21.15.167:3300/0 conn(0x7f811c0b2ee0 msgr2=0x7f811c144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 1 --2- 172.21.15.64:0/1188108462 >> v2:172.21.15.167:3300/0 conn(0x7f811c0b2ee0 0x7f811c144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 1 -- 172.21.15.64:0/1188108462 >> v1:172.21.15.64:6789/0 conn(0x7f811c0b37b0 legacy=0x7f811c0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.385+0000 7f811b7fe640 1 -- 172.21.15.64:0/1188108462 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c06f200 con 0x7f811c150200 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.386+0000 7f947d7a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.386+0000 7f947d7a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7f947d7a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 1 -- 172.21.15.64:0/840354360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec0afc70 msgr2=0x7febec15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 1 --2- 172.21.15.64:0/840354360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec0afc70 0x7febec15f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 1 -- 172.21.15.64:0/840354360 >> v1:172.21.15.167:6789/0 conn(0x7febec157b60 legacy=0x7febec0af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.387+0000 7febeaffd640 1 -- 172.21.15.64:0/840354360 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec0ab490 con 0x7febec073130 2024-04-29T20:23:00.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.388+0000 7f8969030640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.388+0000 7f8969030640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.388+0000 7f8969030640 10 monclient: monmap: 2024-04-29T20:23:00.414 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.389812+0000 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.389812+0000 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.389+0000 7f8969030640 1 Processor -- start 2024-04-29T20:23:00.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.389+0000 7f8969030640 1 -- start start 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.389+0000 7f8969030640 10 monclient: init 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f89640b2230 0x7f8964146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): picked mon.noname-c con 0x7f89640b2230 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89640aa460 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f89640b5c60 0x7f896414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): picked mon.noname-a con 0x7f89640b5c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8964146eb0 con 0x7f89640aa460 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f89640ac1f0 con 0x7f89640b5c60 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f89640ac4d0 con 0x7f89640b2230 2024-04-29T20:23:00.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.390+0000 7f8969030640 10 monclient(hunting): authenticate will time out at 1529.550139s 2024-04-29T20:23:00.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.391+0000 7f811b7fe640 1 --2- 172.21.15.64:0/1188108462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c150200 0x7f811c1525f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f811c06ee50 tx=0x7f811002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1adaca87b9189c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 1 -- 172.21.15.64:0/1125711392 >> v1:172.21.15.167:6789/0 conn(0x7f947813c910 legacy=0x7f9478149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 1 -- 172.21.15.64:0/1125711392 >> v1:172.21.15.64:6789/0 conn(0x7f94780abc10 legacy=0x7f94780adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.392+0000 7f947d7a5640 1 -- 172.21.15.64:0/1125711392 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94780ad6d0 con 0x7f94780ab1a0 2024-04-29T20:23:00.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f37377fe640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3730032c60 0x7f37300c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7fb7ed6f3640 1 --2- 172.21.15.64:0/2629760342 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8144a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009dc0 tx=0x7fb7e002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=519e44a614e2244d server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 1 -- 172.21.15.64:0/1188108462 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81100315a0 con 0x7f811c150200 2024-04-29T20:23:00.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 10 monclient: dump: 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f8119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f81218e4640 5 monclient: authenticate success, global_id 4287 2024-04-29T20:23:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.394+0000 7f81218e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7febeaffd640 1 --2- 172.21.15.64:0/840354360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 0x7febec0aaf50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febdc0097a0 tx=0x7febdc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffde74bc4c0e0c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f37377fe640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3730032c60 0x7f37300c1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34454/0 (socket says 172.21.15.64:34454) 2024-04-29T20:23:00.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f37377fe640 1 -- 172.21.15.64:0/2080554689 learned_addr learned my addr 172.21.15.64:0/2080554689 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f37377fe640 10 monclient(hunting): get_auth_request con 0x7f3730032c60 auth_method 0 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f37377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f37377fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f37377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.395+0000 7f3efad9d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f3efc13c910 0x7f3efc0ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f37377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f37377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f37377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f3efad9d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f3efc13c910 0x7f3efc0ae390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33448/0 (socket says 172.21.15.64:33448) 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f3efad9d640 1 -- 172.21.15.64:0/2334915479 learned_addr learned my addr 172.21.15.64:0/2334915479 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f3efad9d640 10 monclient(hunting): get_auth_request con 0x7f3efc13c910 auth_method 0 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f3efad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f3efad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.397+0000 7f3efad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.398+0000 7f8962d9d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f89640b2230 0x7f8964146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f3737fff640 1 --2- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f37300c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f3efad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f3efad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f3efad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f28f6ffd640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f28f80aab10 0x7f28f80aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f8119ffb640 1 -- 172.21.15.64:0/1188108462 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8110037070 con 0x7f811c150200 2024-04-29T20:23:00.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f8962d9d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f89640b2230 0x7f8964146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33464/0 (socket says 172.21.15.64:33464) 2024-04-29T20:23:00.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8962d9d640 1 -- 172.21.15.64:0/429084537 learned_addr learned my addr 172.21.15.64:0/429084537 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f947d7a5640 1 --2- 172.21.15.64:0/1125711392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 0x7f947813c3d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f94680098d0 tx=0x7f946802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e73c6c90c6cf56 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8962d9d640 10 monclient(hunting): get_auth_request con 0x7f89640b2230 auth_method 0 2024-04-29T20:23:00.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8962d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8962d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8962d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f8119ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f81218e4640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.399+0000 7f81218e4640 4 set_mon_vals no callback set 2024-04-29T20:23:00.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f28f6ffd640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f28f80aab10 0x7f28f80aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33454/0 (socket says 172.21.15.64:33454) 2024-04-29T20:23:00.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f28f6ffd640 1 -- 172.21.15.64:0/2945276798 learned_addr learned my addr 172.21.15.64:0/2945276798 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f28f6ffd640 10 monclient(hunting): get_auth_request con 0x7f28f80aab10 auth_method 0 2024-04-29T20:23:00.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f28f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f28f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f28f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8119ffb640 1 -- 172.21.15.64:0/1188108462 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110031960 con 0x7f811c150200 2024-04-29T20:23:00.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8119ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f8119ffb640 10 monclient: dump: 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 10 monclient: shutdown 2024-04-29T20:23:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 1 -- 172.21.15.64:0/1188108462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c150200 msgr2=0x7f811c1525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 1 --2- 172.21.15.64:0/1188108462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c150200 0x7f811c1525f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f811c06ee50 tx=0x7f811002b840 comp rx=0 tx=0).stop 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 1 -- 172.21.15.64:0/1188108462 shutdown_connections 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 1 --2- 172.21.15.64:0/1188108462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c150200 0x7f811c1525f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 1 --2- 172.21.15.64:0/1188108462 >> v2:172.21.15.167:3300/0 conn(0x7f811c0b2ee0 0x7f811c144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.400+0000 7f81218e4640 1 -- 172.21.15.64:0/1188108462 >> 172.21.15.64:0/1188108462 conn(0x7f811c06c960 msgr2=0x7f811c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f81218e4640 1 -- 172.21.15.64:0/1188108462 shutdown_connections 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f81218e4640 1 -- 172.21.15.64:0/1188108462 wait complete. 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f3737fff640 10 monclient(hunting): get_auth_request con 0x7f3730006520 auth_method 0 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f3737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f3737fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f3737fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 1 -- 172.21.15.64:0/840354360 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc035020 con 0x7febec073130 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: dump: 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.432 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 1 -- 172.21.15.64:0/840354360 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febdc031830 con 0x7febec073130 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 1 -- 172.21.15.64:0/840354360 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc031c40 con 0x7febec073130 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febe9ffb640 10 monclient: dump: 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febf1406640 5 monclient: authenticate success, global_id 4303 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febf1406640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7febf1406640 4 set_mon_vals no callback set 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 1 -- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 msgr2=0x7f37300c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 1 --2- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f37300c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 1 -- 172.21.15.64:0/2080554689 >> v2:172.21.15.167:3300/0 conn(0x7f37300cb270 msgr2=0x7f37300cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 1 --2- 172.21.15.64:0/2080554689 >> v2:172.21.15.167:3300/0 conn(0x7f37300cb270 0x7f37300cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.401+0000 7f37377fe640 1 -- 172.21.15.64:0/2080554689 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3730031f80 con 0x7f3730032c60 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 10 monclient: monmap: 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.436 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 10 monclient: shutdown 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 -- 172.21.15.64:0/840354360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 msgr2=0x7febec0aaf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 --2- 172.21.15.64:0/840354360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 0x7febec0aaf50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febdc0097a0 tx=0x7febdc02b8d0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 1 librados: starting msgr at 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 1 librados: starting objecter 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 -- 172.21.15.64:0/840354360 shutdown_connections 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 --2- 172.21.15.64:0/840354360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec0afc70 0x7febec15f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 --2- 172.21.15.64:0/840354360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 0x7febec0aaf50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 -- 172.21.15.64:0/840354360 >> 172.21.15.64:0/840354360 conn(0x7febec0b4b60 msgr2=0x7febec0b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 -- 172.21.15.64:0/840354360 shutdown_connections 2024-04-29T20:23:00.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7febf1406640 1 -- 172.21.15.64:0/840354360 wait complete. 2024-04-29T20:23:00.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 1 Processor -- start 2024-04-29T20:23:00.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 1 -- start start 2024-04-29T20:23:00.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 1 librados: setting wanted keys 2024-04-29T20:23:00.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 1 librados: calling monclient init 2024-04-29T20:23:00.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f81218e4640 10 monclient: init 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7febf1406640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7febf1406640 10 monclient: monmap: 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.440 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 1 -- 172.21.15.64:0/2334915479 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc0ab1a0 msgr2=0x7f3efc13c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 1 --2- 172.21.15.64:0/2334915479 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc0ab1a0 0x7f3efc13c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 1 -- 172.21.15.64:0/2334915479 >> v1:172.21.15.64:6789/0 conn(0x7f3efc0abc10 legacy=0x7f3efc0adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.402+0000 7f3efad9d640 1 -- 172.21.15.64:0/2334915479 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc0ad3f0 con 0x7f3efc13c910 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f811c0b2ee0 0x7f811c13fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): picked mon.noname-c con 0x7f811c0b2ee0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c0b37b0 0x7f811c13bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): picked mon.noname-a con 0x7f811c0b37b0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f811c150200 0x7f811c13c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): picked mon.noname-b con 0x7f811c150200 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f811c148730 con 0x7f811c0b37b0 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1482d0 con 0x7f811c150200 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f811c147de0 con 0x7f811c0b2ee0 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.403+0000 7f81218e4640 10 monclient(hunting): authenticate will time out at 1529.563437s 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 1 librados: starting msgr at 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 1 librados: starting objecter 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 1 Processor -- start 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 1 -- start start 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 1 librados: setting wanted keys 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 1 librados: calling monclient init 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7febf1406640 10 monclient: init 2024-04-29T20:23:00.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.404+0000 7f28f77fe640 1 --2- 172.21.15.64:0/2945276798 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f814b890 0x7f28f81477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 0x7febec0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): picked mon.noname-b con 0x7febec073130 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febec0afc70 0x7febec14bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): picked mon.noname-a con 0x7febec0afc70 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec157b60 0x7febec14e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): picked mon.noname-c con 0x7febec157b60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7febec0af170 con 0x7febec0afc70 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7f28f67fc640 1 --2- 172.21.15.64:0/2945276798 >> v2:172.21.15.64:3300/0 conn(0x7f28f80ad460 0x7f28f814b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7febec0aa700 con 0x7febec073130 2024-04-29T20:23:00.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7febec157fa0 con 0x7febec157b60 2024-04-29T20:23:00.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.405+0000 7febf1406640 10 monclient(hunting): authenticate will time out at 1529.564699s 2024-04-29T20:23:00.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f37377fe640 1 --2- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3730032c60 0x7f37300c1c30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3730031460 tx=0x7f372402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234c788555c78cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f28f67fc640 10 monclient(hunting): get_auth_request con 0x7f28f80ad460 auth_method 0 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f28f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f28f67fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f28f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f8962d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f8962d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.406+0000 7f8962d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.407+0000 7f28f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.407+0000 7f28f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.407+0000 7f28f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f811b7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f811c150200 0x7f811c13c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 1 -- 172.21.15.64:0/1125711392 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468038070 con 0x7f94780ab1a0 2024-04-29T20:23:00.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: dump: 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 1 -- 172.21.15.64:0/1125711392 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9468031710 con 0x7f94780ab1a0 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f947fa09640 5 monclient: authenticate success, global_id 4232 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f947fa09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f947fa09640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f947fa09640 4 set_mon_vals no callback set 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 1 -- 172.21.15.64:0/1125711392 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468031ae0 con 0x7f94780ab1a0 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.409+0000 7f946ffff640 10 monclient: dump: 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 10 monclient: shutdown 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f28f77fe640 10 monclient(hunting): get_auth_request con 0x7f28f814b890 auth_method 0 2024-04-29T20:23:00.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f28f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f28f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f28f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 -- 172.21.15.64:0/1125711392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 msgr2=0x7f947813c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 --2- 172.21.15.64:0/1125711392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 0x7f947813c3d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f94680098d0 tx=0x7f946802b7b0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 -- 172.21.15.64:0/1125711392 shutdown_connections 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 --2- 172.21.15.64:0/1125711392 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780ab1a0 0x7f947813c3d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 -- 172.21.15.64:0/1125711392 >> 172.21.15.64:0/1125711392 conn(0x7f94780b4b60 msgr2=0x7f94780b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 -- 172.21.15.64:0/1125711392 shutdown_connections 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.410+0000 7f947fa09640 1 -- 172.21.15.64:0/1125711392 wait complete. 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f28f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f28f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f28f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-04-29T20:23:00.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 1 -- 172.21.15.64:0/429084537 >> v2:172.21.15.64:3300/0 conn(0x7f89640b5c60 msgr2=0x7f896414dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 1 --2- 172.21.15.64:0/429084537 >> v2:172.21.15.64:3300/0 conn(0x7f89640b5c60 0x7f896414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 1 -- 172.21.15.64:0/429084537 >> v1:172.21.15.64:6789/0 conn(0x7f89640aa460 legacy=0x7f89640b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f8962d9d640 1 -- 172.21.15.64:0/429084537 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89640aa8d0 con 0x7f89640b2230 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 1 -- 172.21.15.64:0/2629760342 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0039070 con 0x7fb7e80aadd0 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: dump: 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.457 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 1 -- 172.21.15.64:0/2629760342 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e002cb20 con 0x7fb7e80aadd0 2024-04-29T20:23:00.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f947fa09640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f947fa09640 10 monclient: monmap: 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.459 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 1 -- 172.21.15.64:0/2629760342 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0034440 con 0x7fb7e80aadd0 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7dffff640 10 monclient: dump: 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.461 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.461 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.461 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f947fa09640 1 librados: starting msgr at 2024-04-29T20:23:00.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7f947fa09640 1 librados: starting objecter 2024-04-29T20:23:00.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f947fa09640 1 Processor -- start 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f3efad9d640 1 --2- 172.21.15.64:0/2334915479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc13c910 0x7f3efc0ae390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c310 tx=0x7f3ef402d1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38a8e1cbcc5d6663 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f811b7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f811c150200 0x7f811c13c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33472/0 (socket says 172.21.15.64:33472) 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f811b7fe640 1 -- 172.21.15.64:0/3599402015 learned_addr learned my addr 172.21.15.64:0/3599402015 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f811b7fe640 10 monclient(hunting): get_auth_request con 0x7f811c150200 auth_method 0 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f811b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f811b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f811b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7ef957640 5 monclient: authenticate success, global_id 4226 2024-04-29T20:23:00.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7ef957640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.411+0000 7fb7ef957640 4 set_mon_vals no callback set 2024-04-29T20:23:00.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f947fa09640 1 -- start start 2024-04-29T20:23:00.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f947fa09640 1 librados: setting wanted keys 2024-04-29T20:23:00.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f947fa09640 1 librados: calling monclient init 2024-04-29T20:23:00.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7f947fa09640 10 monclient: init 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 10 monclient: shutdown 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 -- 172.21.15.64:0/2629760342 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 msgr2=0x7fb7e8144a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 --2- 172.21.15.64:0/2629760342 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8144a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009dc0 tx=0x7fb7e002bd70 comp rx=0 tx=0).stop 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 -- 172.21.15.64:0/2629760342 shutdown_connections 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 --2- 172.21.15.64:0/2629760342 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8144a50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 -- 172.21.15.64:0/2629760342 >> 172.21.15.64:0/2629760342 conn(0x7fb7e80b4f80 msgr2=0x7fb7e80b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 -- 172.21.15.64:0/2629760342 shutdown_connections 2024-04-29T20:23:00.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.412+0000 7fb7ef957640 1 -- 172.21.15.64:0/2629760342 wait complete. 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94780ab1a0 0x7f94780b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): picked mon.noname-a con 0x7f94780ab1a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780abc10 0x7f9478070410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): picked mon.noname-c con 0x7f94780abc10 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947813c910 0x7f9478070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): picked mon.noname-b con 0x7f947813c910 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f94780b5680 con 0x7f94780ab1a0 2024-04-29T20:23:00.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f94780ac160 con 0x7f947813c910 2024-04-29T20:23:00.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f94780b5800 con 0x7f94780abc10 2024-04-29T20:23:00.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 1 -- 172.21.15.64:0/2945276798 >> v2:172.21.15.167:3300/0 conn(0x7f28f80aab10 msgr2=0x7f28f80aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 1 --2- 172.21.15.64:0/2945276798 >> v2:172.21.15.167:3300/0 conn(0x7f28f80aab10 0x7f28f80aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 1 -- 172.21.15.64:0/2945276798 >> v2:172.21.15.64:3300/0 conn(0x7f28f80ad460 msgr2=0x7f28f814b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 1 --2- 172.21.15.64:0/2945276798 >> v2:172.21.15.64:3300/0 conn(0x7f28f80ad460 0x7f28f814b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7fb7ef957640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f28f77fe640 1 -- 172.21.15.64:0/2945276798 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f814ae30 con 0x7f28f814b890 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7fb7ef957640 10 monclient: monmap: 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.469 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.470 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.470 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.470 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f811b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f947fa09640 10 monclient(hunting): authenticate will time out at 1529.572671s 2024-04-29T20:23:00.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f811b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.413+0000 7f811b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 1 librados: starting msgr at 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 1 librados: starting objecter 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 1 Processor -- start 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 1 -- start start 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 1 librados: setting wanted keys 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 1 librados: calling monclient init 2024-04-29T20:23:00.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.414+0000 7fb7ef957640 10 monclient: init 2024-04-29T20:23:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:23:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): picked mon.noname-c con 0x7fb7e80aadd0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7f811a7fc640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c0b37b0 0x7f811c13bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7e80ab1b0 0x7fb7e8070890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e80ab1b0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e80734c0 0x7fb7e8070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): picked mon.noname-a con 0x7fb7e80734c0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e814bd40 con 0x7fb7e80734c0 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e80ad830 con 0x7fb7e80ab1b0 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e81533b0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.415+0000 7fb7ef957640 10 monclient(hunting): authenticate will time out at 1529.574975s 2024-04-29T20:23:00.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.416+0000 7f8962d9d640 1 --2- 172.21.15.64:0/429084537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b2230 0x7f8964146970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89640b9df0 tx=0x7f895402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4733492a5344f3c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.417+0000 7f28f77fe640 1 --2- 172.21.15.64:0/2945276798 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f814b890 0x7f28f81477e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f28f80b5200 tx=0x7f28f002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ca076f1a340f971 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.418+0000 7f811a7fc640 10 monclient(hunting): get_auth_request con 0x7f811c0b37b0 auth_method 0 2024-04-29T20:23:00.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.418+0000 7f811a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.418+0000 7f811a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.418+0000 7f811a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 1 -- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f811c0b2ee0 msgr2=0x7f811c13fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f811c0b2ee0 0x7f811c13fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 1 -- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c0b37b0 msgr2=0x7f811c13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f811c0b37b0 0x7f811c13bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.419+0000 7f811b7fe640 1 -- 172.21.15.64:0/3599402015 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c14a550 con 0x7f811c150200 2024-04-29T20:23:00.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.421+0000 7f947d7a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94780ab1a0 0x7f94780b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.422+0000 7febeb7fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec157b60 0x7febec14e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 1 -- 172.21.15.64:0/2080554689 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724035020 con 0x7f3730032c60 2024-04-29T20:23:00.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: dump: 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 1 -- 172.21.15.64:0/2080554689 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3724031810 con 0x7f3730032c60 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 1 -- 172.21.15.64:0/2080554689 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724031be0 con 0x7f3730032c60 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f3736ffd640 10 monclient: dump: 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.480 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f373d6e8640 5 monclient: authenticate success, global_id 4293 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f373d6e8640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7f373d6e8640 4 set_mon_vals no callback set 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febeb7fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec157b60 0x7febec14e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36670/0 (socket says 172.21.15.64:36670) 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febeb7fe640 1 -- 172.21.15.64:0/2936311155 learned_addr learned my addr 172.21.15.64:0/2936311155 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febeb7fe640 10 monclient(hunting): get_auth_request con 0x7febec157b60 auth_method 0 2024-04-29T20:23:00.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febeb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.423+0000 7febea7fc640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febec0afc70 0x7febec14bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7febea7fc640 10 monclient(hunting): get_auth_request con 0x7febec0afc70 auth_method 0 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7febea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7febea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7febea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 10 monclient: shutdown 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 -- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3730032c60 msgr2=0x7f37300c1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 --2- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3730032c60 0x7f37300c1c30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3730031460 tx=0x7f372402b8b0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 -- 172.21.15.64:0/2080554689 shutdown_connections 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 --2- 172.21.15.64:0/2080554689 >> v2:172.21.15.167:3300/0 conn(0x7f37300cb270 0x7f37300cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 --2- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3730032c60 0x7f37300c1c30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 --2- 172.21.15.64:0/2080554689 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f37300c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 -- 172.21.15.64:0/2080554689 >> 172.21.15.64:0/2080554689 conn(0x7f3730006d20 msgr2=0x7f3730007130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 -- 172.21.15.64:0/2080554689 shutdown_connections 2024-04-29T20:23:00.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f373d6e8640 1 -- 172.21.15.64:0/2080554689 wait complete. 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947cfa4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780abc10 0x7f9478070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947d7a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94780ab1a0 0x7f94780b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34502/0 (socket says 172.21.15.64:34502) 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947d7a5640 1 -- 172.21.15.64:0/1665183110 learned_addr learned my addr 172.21.15.64:0/1665183110 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947d7a5640 10 monclient(hunting): get_auth_request con 0x7f94780ab1a0 auth_method 0 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947d7a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947d7a5640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.424+0000 7f947d7a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7febeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7febeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7febeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 10 monclient: monmap: 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 1 librados: starting msgr at 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 1 librados: starting objecter 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 1 Processor -- start 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 1 -- start start 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 1 librados: setting wanted keys 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 1 librados: calling monclient init 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7febea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7febea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7f373d6e8640 10 monclient: init 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.425+0000 7febea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 1 -- 172.21.15.64:0/2334915479 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400a0d0 con 0x7f3efc13c910 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: dump: 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.488 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 1 -- 172.21.15.64:0/2334915479 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef400a270 con 0x7f3efc13c910 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3f00af3640 5 monclient: authenticate success, global_id 4312 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3f00af3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:23:00.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f373002cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): picked mon.noname-c con 0x7f3730006520 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3f00af3640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3f00af3640 4 set_mon_vals no callback set 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 1 -- 172.21.15.64:0/2334915479 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400a640 con 0x7f3efc13c910 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f3ef959a640 10 monclient: dump: 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.491 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3730032c60 0x7f37300c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): picked mon.noname-b con 0x7f3730032c60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37300cb270 0x7f3730165510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): picked mon.noname-a con 0x7f37300cb270 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f37300cbdc0 con 0x7f37300cb270 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f37300cbc40 con 0x7f3730032c60 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f37300cbf40 con 0x7f3730006520 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f373d6e8640 10 monclient(hunting): authenticate will time out at 1529.586194s 2024-04-29T20:23:00.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f947d7a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.426+0000 7f947d7a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f947d7a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 10 monclient: shutdown 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 -- 172.21.15.64:0/2334915479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc13c910 msgr2=0x7f3efc0ae390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 --2- 172.21.15.64:0/2334915479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc13c910 0x7f3efc0ae390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c310 tx=0x7f3ef402d1d0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 -- 172.21.15.64:0/2334915479 shutdown_connections 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 --2- 172.21.15.64:0/2334915479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc13c910 0x7f3efc0ae390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 --2- 172.21.15.64:0/2334915479 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc0ab1a0 0x7f3efc13c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 -- 172.21.15.64:0/2334915479 >> 172.21.15.64:0/2334915479 conn(0x7f3efc0b4b60 msgr2=0x7f3efc0b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7fb7edef4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e80734c0 0x7fb7e8070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 -- 172.21.15.64:0/2334915479 shutdown_connections 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.427+0000 7f3f00af3640 1 -- 172.21.15.64:0/2334915479 wait complete. 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 10 monclient: monmap: 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 1 -- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 msgr2=0x7febec0b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febec073130 0x7febec0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 1 -- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febec0afc70 msgr2=0x7febec14bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febec0afc70 0x7febec14bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7febeb7fe640 1 -- 172.21.15.64:0/2936311155 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec0b37f0 con 0x7febec157b60 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 1 librados: starting msgr at 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 1 librados: starting objecter 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 1 Processor -- start 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 1 -- start start 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 1 librados: setting wanted keys 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 1 librados: calling monclient init 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.428+0000 7f3f00af3640 10 monclient: init 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:23:00.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0ab1a0 0x7f3efc1c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): picked mon.noname-a con 0x7f3efc0ab1a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc0abc10 0x7f3efc0b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): picked mon.noname-b con 0x7f3efc0abc10 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc13c910 0x7f3efc1dc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): picked mon.noname-c con 0x7f3efc13c910 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc13cda0 con 0x7f3efc0ab1a0 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc0ae8d0 con 0x7f3efc0abc10 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc0b5680 con 0x7f3efc13c910 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.429+0000 7f3f00af3640 10 monclient(hunting): authenticate will time out at 1529.588937s 2024-04-29T20:23:00.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 1 -- 172.21.15.64:0/2945276798 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0039070 con 0x7f28f814b890 2024-04-29T20:23:00.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: dump: 2024-04-29T20:23:00.501 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 1 -- 172.21.15.64:0/2945276798 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f002cb40 con 0x7f28f814b890 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 1 -- 172.21.15.64:0/2945276798 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0034440 con 0x7f28f814b890 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f7fff640 5 monclient: authenticate success, global_id 4238 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f7fff640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f7fff640 4 set_mon_vals no callback set 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.430+0000 7f28f5ffb640 10 monclient: dump: 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.503 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7fb7edef4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e80734c0 0x7fb7e8070eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34510/0 (socket says 172.21.15.64:34510) 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7fb7edef4640 1 -- 172.21.15.64:0/2627217581 learned_addr learned my addr 172.21.15.64:0/2627217581 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7fb7edef4640 10 monclient(hunting): get_auth_request con 0x7fb7e80734c0 auth_method 0 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7fb7edef4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7fb7edef4640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7fb7edef4640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 10 monclient: shutdown 2024-04-29T20:23:00.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 -- 172.21.15.64:0/2945276798 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f814b890 msgr2=0x7f28f81477e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 --2- 172.21.15.64:0/2945276798 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f814b890 0x7f28f81477e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f28f80b5200 tx=0x7f28f002bd90 comp rx=0 tx=0).stop 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 -- 172.21.15.64:0/2945276798 shutdown_connections 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 --2- 172.21.15.64:0/2945276798 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f814b890 0x7f28f81477e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 --2- 172.21.15.64:0/2945276798 >> v2:172.21.15.64:3300/0 conn(0x7f28f80ad460 0x7f28f814b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 --2- 172.21.15.64:0/2945276798 >> v2:172.21.15.167:3300/0 conn(0x7f28f80aab10 0x7f28f80aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 -- 172.21.15.64:0/2945276798 >> 172.21.15.64:0/2945276798 conn(0x7f28f8068320 msgr2=0x7f28f80b4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 -- 172.21.15.64:0/2945276798 shutdown_connections 2024-04-29T20:23:00.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f28f7fff640 1 -- 172.21.15.64:0/2945276798 wait complete. 2024-04-29T20:23:00.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f947cfa4640 10 monclient(hunting): get_auth_request con 0x7f94780abc10 auth_method 0 2024-04-29T20:23:00.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f947cfa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f947cfa4640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.431+0000 7f947cfa4640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 10 monclient: monmap: 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 1 librados: starting msgr at 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 1 librados: starting objecter 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 1 Processor -- start 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 1 -- start start 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 1 librados: setting wanted keys 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 1 librados: calling monclient init 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.432+0000 7f28f7fff640 10 monclient: init 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:23:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28f80aab10 0x7f28f80b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 1 -- 172.21.15.64:0/429084537 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954035020 con 0x7f89640b2230 2024-04-29T20:23:00.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: dump: 2024-04-29T20:23:00.510 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.513 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.513 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.513 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.513 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 1 -- 172.21.15.64:0/429084537 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8954031870 con 0x7f89640b2230 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 1 -- 172.21.15.64:0/429084537 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954031c40 con 0x7f89640b2230 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f896259c640 10 monclient: dump: 2024-04-29T20:23:00.514 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.515 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f28f80aab10 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f80ad460 0x7f28f80707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f28f80ad460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28f814b890 0x7f28f80729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f28f814b890 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f28f81480f0 con 0x7f28f814b890 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8147f70 con 0x7f28f80aab10 2024-04-29T20:23:00.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f28f8148270 con 0x7f28f80ad460 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f28f7fff640 10 monclient(hunting): authenticate will time out at 1529.593309s 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f8969030640 5 monclient: authenticate success, global_id 4315 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f8969030640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f8969030640 4 set_mon_vals no callback set 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f947cfa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f947cfa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.433+0000 7f947cfa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 10 monclient: shutdown 2024-04-29T20:23:00.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 -- 172.21.15.64:0/429084537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b2230 msgr2=0x7f8964146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 --2- 172.21.15.64:0/429084537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b2230 0x7f8964146970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89640b9df0 tx=0x7f895402b910 comp rx=0 tx=0).stop 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f811b7fe640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f811c150200 0x7f811c13c440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8110034040 tx=0x7f81100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8103fff640 10 client.?.objecter ms_handle_connect 0x7f811c150200 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8103fff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 -- 172.21.15.64:0/429084537 shutdown_connections 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 --2- 172.21.15.64:0/429084537 >> v2:172.21.15.64:3300/0 conn(0x7f89640b5c60 0x7f896414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 --2- 172.21.15.64:0/429084537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b2230 0x7f8964146970 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7fb7ed6f3640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 -- 172.21.15.64:0/429084537 >> 172.21.15.64:0/429084537 conn(0x7f896406c960 msgr2=0x7f896406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 -- 172.21.15.64:0/429084537 shutdown_connections 2024-04-29T20:23:00.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.434+0000 7f8969030640 1 -- 172.21.15.64:0/429084537 wait complete. 2024-04-29T20:23:00.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7febd37fe640 10 client.?.objecter ms_handle_connect 0x7febec157b60 2024-04-29T20:23:00.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7febd37fe640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7febeb7fe640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec157b60 0x7febec14e5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7febe400efc0 tx=0x7febe400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 1 -- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780abc10 msgr2=0x7f9478070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f94780abc10 0x7f9478070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 1 -- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947813c910 msgr2=0x7f9478070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947813c910 0x7f9478070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f947d7a5640 1 -- 172.21.15.64:0/1665183110 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947806fa30 con 0x7f94780ab1a0 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f8969030640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.435+0000 7f8969030640 10 monclient: monmap: 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.523 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7fb7ed6f3640 10 monclient(hunting): get_auth_request con 0x7fb7e80aadd0 auth_method 0 2024-04-29T20:23:00.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7fb7ed6f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7fb7ed6f3640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7fb7ed6f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 1 librados: starting msgr at 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 1 librados: starting objecter 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 1 Processor -- start 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 1 -- start start 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 1 librados: setting wanted keys 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 1 librados: calling monclient init 2024-04-29T20:23:00.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.436+0000 7f8969030640 10 monclient: init 2024-04-29T20:23:00.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89640aa460 0x7f8964143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): picked mon.noname-a con 0x7f89640aa460 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f89640b2230 0x7f89640ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): picked mon.noname-c con 0x7f89640b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b5c60 0x7f89641d8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): picked mon.noname-b con 0x7f89640b5c60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f89640b6210 con 0x7f89640aa460 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f89640b6090 con 0x7f89640b5c60 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f896414e220 con 0x7f89640b2230 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.437+0000 7f8969030640 10 monclient(hunting): authenticate will time out at 1529.597174s 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.438+0000 7fb7edef4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.438+0000 7fb7edef4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.438+0000 7fb7edef4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ecef2640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7e80ab1b0 0x7fb7e8070890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ecef2640 10 monclient(hunting): get_auth_request con 0x7fb7e80ab1b0 auth_method 0 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ecef2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ecef2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ecef2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 1 -- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7e80ab1b0 msgr2=0x7fb7e8070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7e80ab1b0 0x7fb7e8070890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 1 -- 172.21.15.64:0/2627217581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e80734c0 msgr2=0x7fb7e8070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e80734c0 0x7fb7e8070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.440+0000 7fb7ed6f3640 1 -- 172.21.15.64:0/2627217581 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e8072700 con 0x7fb7e80aadd0 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.441+0000 7f37377fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3730032c60 0x7f37300c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.442+0000 7f3737fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f373002cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.442+0000 7f3ef9d9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc0abc10 0x7f3efc0b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f3ef9d9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc0abc10 0x7f3efc0b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33532/0 (socket says 172.21.15.64:33532) 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f3ef9d9b640 1 -- 172.21.15.64:0/1017661381 learned_addr learned my addr 172.21.15.64:0/1017661381 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f3ef9d9b640 10 monclient(hunting): get_auth_request con 0x7f3efc0abc10 auth_method 0 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f3ef9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f3ef9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f3ef9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3730032c60 0x7f37300c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33518/0 (socket says 172.21.15.64:33518) 2024-04-29T20:23:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 1 -- 172.21.15.64:0/3896767166 learned_addr learned my addr 172.21.15.64:0/3896767166 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): get_auth_request con 0x7f3730032c60 auth_method 0 2024-04-29T20:23:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.443+0000 7f37377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.444+0000 7f3ef9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.444+0000 7f3ef9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.444+0000 7f3ef9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.445+0000 7f3efad9d640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc13c910 0x7f3efc1dc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.445+0000 7f947d7a5640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94780ab1a0 0x7f94780b44c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f946802b7f0 tx=0x7f94680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.446+0000 7f946dffb640 10 client.?.objecter ms_handle_connect 0x7f94780ab1a0 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.446+0000 7f946dffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3737fff640 10 monclient(hunting): get_auth_request con 0x7f3730006520 auth_method 0 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3737fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3737fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3efad9d640 10 monclient(hunting): get_auth_request con 0x7f3efc13c910 auth_method 0 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3efad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3efad9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f3efad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7fb7ed6f3640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8147a90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80b1570 tx=0x7fb7e0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7fb7ddffb640 10 client.?.objecter ms_handle_connect 0x7fb7e80aadd0 2024-04-29T20:23:00.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7fb7ddffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 1 -- 172.21.15.64:0/3896767166 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 msgr2=0x7f373002cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3730006520 0x7f373002cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 1 -- 172.21.15.64:0/3896767166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37300cb270 msgr2=0x7f3730165510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37300cb270 0x7f3730165510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.447+0000 7f37377fe640 1 -- 172.21.15.64:0/3896767166 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373002e160 con 0x7f3730032c60 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110032630 con 0x7f811c150200 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: dump: 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.538 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81100327f0 con 0x7f811c150200 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110032be0 con 0x7f811c150200 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f8103fff640 10 monclient: dump: 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.539 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 5 monclient: authenticate success, global_id 4321 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 10 monclient: _renew_subs 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811c148ff0 con 0x7f811c150200 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81197fa640 4 set_mon_vals no callback set 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 10 monclient: _renew_subs 2024-04-29T20:23:00.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811c143200 con 0x7f811c150200 2024-04-29T20:23:00.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f81218e4640 1 librados: init done 2024-04-29T20:23:00.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f3efad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f3efad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.448+0000 7f3efad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7f81218e4640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7f81218e4640 15 client.4321.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7f81218e4640 20 client.4321.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7f81218e4640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7f81218e4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f811c04fce0 con 0x7f811c150200 2024-04-29T20:23:00.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4018020 con 0x7febec157b60 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: dump: 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 5 monclient: authenticate success, global_id 4241 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 10 monclient: _renew_subs 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febec0b3ad0 con 0x7febec157b60 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 10 monclient: _renew_subs 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febec14eb00 con 0x7febec157b60 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febf1406640 1 librados: init done 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe4014480 con 0x7febec157b60 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4018020 con 0x7febec157b60 2024-04-29T20:23:00.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febe97fa640 4 set_mon_vals no callback set 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.449+0000 7febd37fe640 10 monclient: dump: 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.546 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.450+0000 7febf1406640 10 client.4241.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.450+0000 7febf1406640 15 client.4241.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.450+0000 7febf1406640 20 client.4241.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.450+0000 7febf1406640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.450+0000 7febf1406640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.450+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7febcc002910 con 0x7febec157b60 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.451+0000 7f28f6ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28f80aab10 0x7f28f80b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f28f77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28f814b890 0x7f28f80729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 1 -- 172.21.15.64:0/1017661381 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc13c910 msgr2=0x7f3efc1dc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3efc13c910 0x7f3efc1dc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 1 -- 172.21.15.64:0/1017661381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0ab1a0 msgr2=0x7f3efc1c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0ab1a0 0x7f3efc1c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.453+0000 7f3ef9d9b640 1 -- 172.21.15.64:0/1017661381 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc1dbb90 con 0x7f3efc0abc10 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468043070 con 0x7f94780ab1a0 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: dump: 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.550 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9468033950 con 0x7f94780ab1a0 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 5 monclient: authenticate success, global_id 4305 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 10 monclient: _renew_subs 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 1 -- 172.21.15.64:0/1665183110 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947806fd10 con 0x7f94780ab1a0 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468038040 con 0x7f94780ab1a0 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946dffb640 10 monclient: dump: 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 10 monclient: _renew_subs 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 1 -- 172.21.15.64:0/1665183110 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9478139800 con 0x7f94780ab1a0 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f947fa09640 1 librados: init done 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.454+0000 7f946f7fe640 4 set_mon_vals no callback set 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28f814b890 0x7f28f80729c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34536/0 (socket says 172.21.15.64:34536) 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f77fe640 1 -- 172.21.15.64:0/1228931810 learned_addr learned my addr 172.21.15.64:0/1228931810 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f77fe640 10 monclient(hunting): get_auth_request con 0x7f28f814b890 auth_method 0 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f37377fe640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3730032c60 0x7f37300c7f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3724009960 tx=0x7f37240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f3734ff9640 10 client.?.objecter ms_handle_connect 0x7f3730032c60 2024-04-29T20:23:00.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f3734ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f6ffd640 10 monclient(hunting): get_auth_request con 0x7f28f80aab10 auth_method 0 2024-04-29T20:23:00.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f947fa09640 10 client.4305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f28f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f947fa09640 15 client.4305.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f947fa09640 20 client.4305.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f947fa09640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f947fa09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.455+0000 7f947fa09640 1 -- 172.21.15.64:0/1665183110 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9478070100 con 0x7f94780ab1a0 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.456+0000 7f28f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.456+0000 7f28f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.456+0000 7f28f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.457+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.457+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.457+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.458+0000 7f3ef9d9b640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc0abc10 0x7f3efc0b4510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c440 tx=0x7f3ef000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.458+0000 7f3eeaffd640 10 client.?.objecter ms_handle_connect 0x7f3efc0abc10 2024-04-29T20:23:00.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.458+0000 7f3eeaffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.459+0000 7f896359e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b5c60 0x7f89641d8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.463+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724040070 con 0x7f3730032c60 2024-04-29T20:23:00.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f896359e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b5c60 0x7f89641d8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33550/0 (socket says 172.21.15.64:33550) 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: dump: 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.559 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f896359e640 1 -- 172.21.15.64:0/704879801 learned_addr learned my addr 172.21.15.64:0/704879801 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f896359e640 10 monclient(hunting): get_auth_request con 0x7f89640b5c60 auth_method 0 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f896359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f896359e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f896359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 5 monclient: authenticate success, global_id 4327 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 10 monclient: _renew_subs 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37300c8460 con 0x7f3730032c60 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3724039cf0 con 0x7f3730032c60 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724035040 con 0x7f3730032c60 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f3734ff9640 10 monclient: dump: 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 10 monclient: _renew_subs 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3730165a50 con 0x7f3730032c60 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f373d6e8640 1 librados: init done 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.464+0000 7f37367fc640 4 set_mon_vals no callback set 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.465+0000 7f373d6e8640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.465+0000 7f373d6e8640 15 client.4327.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.465+0000 7f373d6e8640 20 client.4327.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.465+0000 7f373d6e8640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.465+0000 7f373d6e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.465+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3730005710 con 0x7f3730032c60 2024-04-29T20:23:00.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f895a59c640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f89640b2230 0x7f89640ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 1 -- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f80ad460 msgr2=0x7f28f80707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f28f80ad460 0x7f28f80707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 1 -- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28f814b890 msgr2=0x7f28f80729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28f814b890 0x7f28f80729c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.466+0000 7f28f6ffd640 1 -- 172.21.15.64:0/1228931810 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f806ff40 con 0x7f28f80aab10 2024-04-29T20:23:00.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.468+0000 7f896359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.468+0000 7f896359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.468+0000 7f896359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-04-29T20:23:00.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 1 -- 172.21.15.64:0/704879801 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f89640b2230 msgr2=0x7f89640ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f89640b2230 0x7f89640ae6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 1 -- 172.21.15.64:0/704879801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89640aa460 msgr2=0x7f8964143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89640aa460 0x7f8964143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.471+0000 7f896359e640 1 -- 172.21.15.64:0/704879801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89640ae0c0 con 0x7f89640b5c60 2024-04-29T20:23:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.486+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febe4005d10 con 0x7febec157b60 2024-04-29T20:23:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.486+0000 7febd37fe640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd806e4b0 0x7febd8070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.487+0000 7febeaffd640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd806e4b0 0x7febd8070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.487+0000 7febeaffd640 10 monclient: get_auth_request con 0x7febd806e4b0 auth_method 0 2024-04-29T20:23:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.487+0000 7febd37fe640 10 client.4241.objecter ms_handle_connect 0x7febd806e4b0 2024-04-29T20:23:00.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.487+0000 7febeaffd640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd806e4b0 0x7febd8070970 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7febdc036000 tx=0x7febdc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.489+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9468046070 con 0x7f94780ab1a0 2024-04-29T20:23:00.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.490+0000 7f946dffb640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f945c06e280 0x7f945c070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.491+0000 7f947cfa4640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f945c06e280 0x7f945c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.491+0000 7f947cfa4640 10 monclient: get_auth_request con 0x7f945c06e280 auth_method 0 2024-04-29T20:23:00.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.491+0000 7f946dffb640 10 client.4305.objecter ms_handle_connect 0x7f945c06e280 2024-04-29T20:23:00.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.491+0000 7f947cfa4640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f945c06e280 0x7f945c070740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f94780b4b10 tx=0x7f9470009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.493+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3724038020 con 0x7f3730032c60 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.493+0000 7f3734ff9640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f371c06e2d0 0x7f371c070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7f3737fff640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f371c06e2d0 0x7f371c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7f3737fff640 10 monclient: get_auth_request con 0x7f371c06e2d0 auth_method 0 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7febe4058b10 con 0x7febec157b60 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7febd37fe640 10 client.4241.objecter ms_dispatch 0x7febec000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7febd37fe640 3 client.4241.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7febd37fe640 3 client.4241.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7f3734ff9640 10 client.4327.objecter ms_handle_connect 0x7f371c06e2d0 2024-04-29T20:23:00.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.494+0000 7f3737fff640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f371c06e2d0 0x7f371c070790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f372c006fd0 tx=0x7f372c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 10 client.4241.objecter checking command 1 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7febd80748f0 0x7febd8076d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 20 client.4241.objecter _get_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 15 client.4241.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 20 client.4241.objecter _get_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 20 client.4241.objecter get_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 15 client.4241.objecter _session_command_op_assign 2 1 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 4 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 10 client.4241.objecter _send_command 1 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 --> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febe401ee50 con 0x7febd80748f0 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 20 client.4241.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7febe4056d10 con 0x7febec157b60 2024-04-29T20:23:00.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febd37fe640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f9468079be0 con 0x7f94780ab1a0 2024-04-29T20:23:00.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 10 client.4305.objecter ms_dispatch 0x7f9478000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 3 client.4305.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7febea7fc640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7febd80748f0 0x7febd8076d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 3 client.4305.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 10 client.4305.objecter checking command 1 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f945c0746c0 0x7f945c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 20 client.4305.objecter _get_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 15 client.4305.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 20 client.4305.objecter _get_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 20 client.4305.objecter get_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 15 client.4305.objecter _session_command_op_assign 3 1 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 4 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 10 client.4305.objecter _send_command 1 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 --> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9468031710 con 0x7f945c0746c0 2024-04-29T20:23:00.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 20 client.4305.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9468077de0 con 0x7f94780ab1a0 2024-04-29T20:23:00.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f946dffb640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7febea7fc640 10 monclient: get_auth_request con 0x7febd80748f0 auth_method 0 2024-04-29T20:23:00.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.495+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f372407a0c0 con 0x7f3730032c60 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 10 client.4327.objecter ms_dispatch 0x7f3738000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 3 client.4327.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 3 client.4327.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 10 client.4327.objecter checking command 1 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7f371c074710 0x7f371c076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 20 client.4327.objecter _get_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 15 client.4327.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 20 client.4327.objecter _get_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 20 client.4327.objecter get_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 15 client.4327.objecter _session_command_op_assign 4 1 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 4 2024-04-29T20:23:00.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 10 client.4327.objecter _send_command 1 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 --> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3724045ec0 con 0x7f371c074710 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 20 client.4327.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f37240782c0 con 0x7f3730032c60 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f3734ff9640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0039050 con 0x7fb7e80aadd0 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: dump: 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e0045070 con 0x7fb7e80aadd0 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ef957640 5 monclient: authenticate success, global_id 4247 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ef957640 10 monclient: _renew_subs 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ef957640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e80729e0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e003e850 con 0x7fb7e80aadd0 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7ddffb640 10 monclient: dump: 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7f373cee7640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7f371c074710 0x7f371c076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.496+0000 7fb7df7fe640 4 set_mon_vals no callback set 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.497+0000 7fb7ef957640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.497+0000 7fb7ef957640 10 monclient: _renew_subs 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.497+0000 7fb7ef957640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.497+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e81e0540 con 0x7fb7e80aadd0 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.497+0000 7fb7ef957640 1 librados: init done 2024-04-29T20:23:00.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.497+0000 7f373cee7640 10 monclient: get_auth_request con 0x7f371c074710 auth_method 0 2024-04-29T20:23:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7fb7ef957640 10 client.4247.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7fb7ef957640 15 client.4247.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7fb7ef957640 20 client.4247.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7fb7ef957640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7fb7ef957640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb7e804fce0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7f896359e640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b5c60 0x7f89641d8950 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f895c00b970 tx=0x7f895c00be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7f895bfff640 10 client.?.objecter ms_handle_connect 0x7f89640b5c60 2024-04-29T20:23:00.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.498+0000 7f895bfff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.500+0000 7febea7fc640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7febd80748f0 0x7febd8076d10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.500+0000 7febd37fe640 10 client.4241.objecter ms_handle_connect 0x7febd80748f0 2024-04-29T20:23:00.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.501+0000 7f3734ff9640 10 client.4327.objecter ms_handle_connect 0x7f371c074710 2024-04-29T20:23:00.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.501+0000 7f373cee7640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7f371c074710 0x7f371c076b30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0017070 con 0x7f3efc0abc10 2024-04-29T20:23:00.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: dump: 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef00035f0 con 0x7f3efc0abc10 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef00039e0 con 0x7f3efc0abc10 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3eeaffd640 10 monclient: dump: 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.586 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3f00af3640 5 monclient: authenticate success, global_id 4330 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3f00af3640 10 monclient: _renew_subs 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3f00af3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc0b2750 con 0x7f3efc0abc10 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.507+0000 7f3ef8d99640 4 set_mon_vals no callback set 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 10 monclient: _renew_subs 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc136d00 con 0x7f3efc0abc10 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 1 librados: init done 2024-04-29T20:23:00.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f28f6ffd640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28f80aab10 0x7f28f80b39b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28e800e7d0 tx=0x7f28e800ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f28dffff640 10 client.?.objecter ms_handle_connect 0x7f28f80aab10 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f28dffff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 15 client.4330.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 20 client.4330.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.508+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3efc04fce0 con 0x7f3efc0abc10 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c019070 con 0x7f89640b5c60 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: dump: 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f895c00cb60 con 0x7f89640b5c60 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c014440 con 0x7f89640b5c60 2024-04-29T20:23:00.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f895bfff640 10 monclient: dump: 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f8961d9b640 4 set_mon_vals no callback set 2024-04-29T20:23:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.509+0000 7f8969030640 5 monclient: authenticate success, global_id 4336 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 10 monclient: _renew_subs 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89640ae350 con 0x7f89640b5c60 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 10 monclient: _renew_subs 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89641d9070 con 0x7f89640b5c60 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.510+0000 7f8969030640 1 librados: init done 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.511+0000 7f8969030640 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.511+0000 7f8969030640 15 client.4336.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.511+0000 7f8969030640 20 client.4336.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.511+0000 7f8969030640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.511+0000 7f8969030640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.511+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f896404fce0 con 0x7f89640b5c60 2024-04-29T20:23:00.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.523+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb7e003e9f0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.523+0000 7fb7ddffb640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb7cc06e280 0x7fb7cc070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.524+0000 7fb7ecef2640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb7cc06e280 0x7fb7cc070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.524+0000 7fb7ecef2640 10 monclient: get_auth_request con 0x7fb7cc06e280 auth_method 0 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.524+0000 7fb7ecef2640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb7cc06e280 0x7fb7cc070740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7e81412e0 tx=0x7fb7e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.524+0000 7fb7ddffb640 10 client.4247.objecter ms_handle_connect 0x7fb7cc06e280 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7fb7e007aee0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 10 client.4247.objecter ms_dispatch 0x7fb7e8000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 3 client.4247.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 3 client.4247.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f811004c020 con 0x7f811c150200 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 10 client.4247.objecter checking command 1 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7fb7cc0746c0 0x7fb7cc076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 20 client.4247.objecter _get_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 15 client.4247.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 20 client.4247.objecter _get_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 20 client.4247.objecter get_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 15 client.4247.objecter _session_command_op_assign 5 1 2024-04-29T20:23:00.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 4 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 10 client.4247.objecter _send_command 1 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 --> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7e0036ec0 con 0x7fb7cc0746c0 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 20 client.4247.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb7e00790e0 con 0x7fb7e80aadd0 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.527+0000 7fb7ddffb640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.528+0000 7f8103fff640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f810806e280 0x7f8108070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.528+0000 7f811affd640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f810806e280 0x7f8108070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.528+0000 7f811affd640 10 monclient: get_auth_request con 0x7f810806e280 auth_method 0 2024-04-29T20:23:00.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.528+0000 7fb7edef4640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7fb7cc0746c0 0x7fb7cc076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.529+0000 7f811affd640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f810806e280 0x7f8108070740 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81140070b0 tx=0x7f8114007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.529+0000 7f8103fff640 10 client.4321.objecter ms_handle_connect 0x7f810806e280 2024-04-29T20:23:00.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.529+0000 7fb7edef4640 10 monclient: get_auth_request con 0x7fb7cc0746c0 auth_method 0 2024-04-29T20:23:00.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f811002f020 con 0x7f811c150200 2024-04-29T20:23:00.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 10 client.4321.objecter ms_dispatch 0x7f811c000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 3 client.4321.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 3 client.4321.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 10 client.4321.objecter checking command 1 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f81080746c0 0x7f8108076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 20 client.4321.objecter _get_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 15 client.4321.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 20 client.4321.objecter _get_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 20 client.4321.objecter get_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 15 client.4321.objecter _session_command_op_assign 0 1 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 4 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 10 client.4321.objecter _send_command 1 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f811004be50 con 0x7f81080746c0 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.530+0000 7f8103fff640 20 client.4321.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.531+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8110077cb0 con 0x7f811c150200 2024-04-29T20:23:00.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.531+0000 7f8103fff640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.531+0000 7f811a7fc640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f81080746c0 0x7f8108076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.531+0000 7f811a7fc640 10 monclient: get_auth_request con 0x7f81080746c0 auth_method 0 2024-04-29T20:23:00.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.532+0000 7fb7edef4640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7fb7cc0746c0 0x7fb7cc076ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.532+0000 7fb7ddffb640 10 client.4247.objecter ms_handle_connect 0x7fb7cc0746c0 2024-04-29T20:23:00.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8017070 con 0x7f28f80aab10 2024-04-29T20:23:00.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: dump: 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e800c940 con 0x7f28f80aab10 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 5 monclient: authenticate success, global_id 4333 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e800cd10 con 0x7f28f80aab10 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28dffff640 10 monclient: dump: 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.604 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 10 monclient: _renew_subs 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f8070cf0 con 0x7f28f80aab10 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f57fa640 4 set_mon_vals no callback set 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 10 monclient: _renew_subs 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f80730e0 con 0x7f28f80aab10 2024-04-29T20:23:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.535+0000 7f28f7fff640 1 librados: init done 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.536+0000 7f28f7fff640 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.536+0000 7f28f7fff640 15 client.4333.objecter _session_command_op_assign -1 1 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.536+0000 7f28f7fff640 20 client.4333.objecter _get_session osd=-1 returning homeless 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.536+0000 7f28f7fff640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.536+0000 7f28f7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.536+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f28f804fce0 con 0x7f28f80aab10 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.537+0000 7f8103fff640 10 client.4321.objecter ms_handle_connect 0x7f81080746c0 2024-04-29T20:23:00.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.537+0000 7f811a7fc640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f81080746c0 0x7f8108076ae0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.541+0000 7f947dfa6640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f945c0746c0 0x7f945c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.547+0000 7f947dfa6640 10 monclient: get_auth_request con 0x7f945c0746c0 auth_method 0 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.549+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== osd.2 v2:172.21.15.64:6800/2373308255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7febe401ee50 con 0x7febd80748f0 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.549+0000 7febd37fe640 10 client.4241.objecter ms_dispatch 0x7febec000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.549+0000 7febd37fe640 10 client.4241.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.549+0000 7febd37fe640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 4 2024-04-29T20:23:00.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.549+0000 7febd37fe640 15 client.4241.objecter _session_command_op_remove 2 1 2024-04-29T20:23:00.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.551+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== osd.4 v2:172.21.15.167:6826/3754201859 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3724045ec0 con 0x7f371c074710 2024-04-29T20:23:00.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.551+0000 7f3734ff9640 10 client.4327.objecter ms_dispatch 0x7f3738000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.551+0000 7f3734ff9640 10 client.4327.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.551+0000 7f3734ff9640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 4 2024-04-29T20:23:00.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.551+0000 7f3734ff9640 15 client.4327.objecter _session_command_op_remove 4 1 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 10 client.4241.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 15 client.4241.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 20 client.4241.objecter _get_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 20 client.4241.objecter _get_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 15 client.4241.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 20 client.4241.objecter get_session s=0x7febd8074220 osd=2 3 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 15 client.4241.objecter _session_command_op_assign 2 2 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 4 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 10 client.4241.objecter _send_command 2 2024-04-29T20:23:00.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.566+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 --> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7febcc001150 con 0x7febd80748f0 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.570+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f28e801a070 con 0x7f28f80aab10 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.570+0000 7f28dffff640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f28e406e2d0 0x7f28e4070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.571+0000 7f28f67fc640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f28e406e2d0 0x7f28e4070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.571+0000 7f28f67fc640 10 monclient: get_auth_request con 0x7f28e406e2d0 auth_method 0 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.571+0000 7f28f67fc640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f28e406e2d0 0x7f28e4070790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006a90 tx=0x7f28ec006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.571+0000 7f28dffff640 10 client.4333.objecter ms_handle_connect 0x7f28e406e2d0 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.572+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f28e8057bc0 con 0x7f28f80aab10 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 10 client.4333.objecter ms_dispatch 0x7f28f8000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 3 client.4333.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 3 client.4333.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 10 client.4333.objecter checking command 1 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7f28e4074710 0x7f28e4076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 20 client.4333.objecter _get_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 15 client.4333.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 20 client.4333.objecter _get_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 20 client.4333.objecter get_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 15 client.4333.objecter _session_command_op_assign 1 1 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 4 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 10 client.4333.objecter _send_command 1 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 --> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28e40771e0 con 0x7f28e4074710 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28dffff640 20 client.4333.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.573+0000 7f28f77fe640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7f28e4074710 0x7f28e4076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.574+0000 7f28f77fe640 10 monclient: get_auth_request con 0x7f28e4074710 auth_method 0 2024-04-29T20:23:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.575+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== osd.5 v2:172.21.15.167:6802/3447508053 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7e0036ec0 con 0x7fb7cc0746c0 2024-04-29T20:23:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.575+0000 7fb7ddffb640 10 client.4247.objecter ms_dispatch 0x7fb7e8000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.575+0000 7fb7ddffb640 10 client.4247.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.575+0000 7fb7ddffb640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 4 2024-04-29T20:23:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.575+0000 7fb7ddffb640 15 client.4247.objecter _session_command_op_remove 5 1 2024-04-29T20:23:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.578+0000 7f947dfa6640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f945c0746c0 0x7f945c076ae0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.578+0000 7f946dffb640 10 client.4305.objecter ms_handle_connect 0x7f945c0746c0 2024-04-29T20:23:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.579+0000 7f28f77fe640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7f28e4074710 0x7f28e4076b30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.579+0000 7f28dffff640 10 client.4333.objecter ms_handle_connect 0x7f28e4074710 2024-04-29T20:23:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febd37fe640 1 -- 172.21.15.64:0/2936311155 <== osd.2 v2:172.21.15.64:6800/2373308255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7febcc001150 con 0x7febd80748f0 2024-04-29T20:23:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febd37fe640 10 client.4241.objecter ms_dispatch 0x7febec000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febd37fe640 10 client.4241.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febd37fe640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 4 2024-04-29T20:23:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febd37fe640 15 client.4241.objecter _session_command_op_remove 2 2 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 10 librados: watch_flush enter 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 10 librados: watch_flush exit 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 10 client.4241.objecter close_session for osd.2 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7febd80748f0 msgr2=0x7febd8076d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7febd80748f0 0x7febd8076d10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 20 client.4241.objecter put_session s=0x7febd8074220 osd=2 1 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 20 client.4241.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.580+0000 7febf1406640 10 client.4241.objecter successfully canceled tick 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd806e4b0 msgr2=0x7febd8070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd806e4b0 0x7febd8070970 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7febdc036000 tx=0x7febdc030000 comp rx=0 tx=0).stop 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 10 monclient: shutdown 2024-04-29T20:23:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec157b60 msgr2=0x7febec14e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 --2- 172.21.15.64:0/2936311155 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febec157b60 0x7febec14e5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7febe400efc0 tx=0x7febe400c490 comp rx=0 tx=0).stop 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febeb7fe640 1 -- 172.21.15.64:0/2936311155 reap_dead start 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 shutdown_connections 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 >> 172.21.15.64:0/2936311155 conn(0x7febec0b4b60 msgr2=0x7febec157350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 shutdown_connections 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 -- 172.21.15.64:0/2936311155 wait complete. 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.581+0000 7febf1406640 1 librados: shutdown 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 15 client.4327.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 20 client.4327.objecter _get_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 20 client.4327.objecter _get_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 15 client.4327.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 20 client.4327.objecter get_session s=0x7f371c074040 osd=4 3 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 15 client.4327.objecter _session_command_op_assign 4 2 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 4 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 10 client.4327.objecter _send_command 2 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.582+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 --> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3730033700 con 0x7f371c074710 2024-04-29T20:23:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.586+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f28e8055dc0 con 0x7f28f80aab10 2024-04-29T20:23:00.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.586+0000 7f28dffff640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.586+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ef001a070 con 0x7f3efc0abc10 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.587+0000 7f3eeaffd640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3edc06e2d0 0x7f3edc070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.587+0000 7f3efa59c640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3edc06e2d0 0x7f3edc070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.587+0000 7f3efa59c640 10 monclient: get_auth_request con 0x7f3edc06e2d0 auth_method 0 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.588+0000 7f3eeaffd640 10 client.4330.objecter ms_handle_connect 0x7f3edc06e2d0 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.588+0000 7f3efa59c640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3edc06e2d0 0x7f3edc070790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009710 tx=0x7f3eec006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f3ef0057460 con 0x7f3efc0abc10 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 10 client.4330.objecter ms_dispatch 0x7f3efc000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 3 client.4330.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 3 client.4330.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 10 client.4330.objecter checking command 1 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7f3edc074710 0x7f3edc076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 20 client.4330.objecter _get_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 15 client.4330.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 20 client.4330.objecter _get_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 20 client.4330.objecter get_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 15 client.4330.objecter _session_command_op_assign 6 1 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 4 2024-04-29T20:23:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 10 client.4330.objecter _send_command 1 2024-04-29T20:23:00.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 --> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3edc0771e0 con 0x7f3edc074710 2024-04-29T20:23:00.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ef0055660 con 0x7f3efc0abc10 2024-04-29T20:23:00.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.590+0000 7f3eeaffd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.591+0000 7f3efad9d640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7f3edc074710 0x7f3edc076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.592+0000 7f3efad9d640 10 monclient: get_auth_request con 0x7f3edc074710 auth_method 0 2024-04-29T20:23:00.623 INFO:teuthology.orchestra.run.smithi064.stdout:64424509446 2024-04-29T20:23:00.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-29T20:23:00.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.597+0000 7f3eeaffd640 10 client.4330.objecter ms_handle_connect 0x7f3edc074710 2024-04-29T20:23:00.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.597+0000 7f3efad9d640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7f3edc074710 0x7f3edc076b30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f3734ff9640 1 -- 172.21.15.64:0/3896767166 <== osd.4 v2:172.21.15.167:6826/3754201859 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3730033700 con 0x7f371c074710 2024-04-29T20:23:00.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f3734ff9640 10 client.4327.objecter ms_dispatch 0x7f3738000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f3734ff9640 10 client.4327.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f3734ff9640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 4 2024-04-29T20:23:00.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f3734ff9640 15 client.4327.objecter _session_command_op_remove 4 2 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 10 client.4247.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 15 client.4247.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 20 client.4247.objecter _get_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 20 client.4247.objecter _get_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 15 client.4247.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 20 client.4247.objecter get_session s=0x7fb7cc073ff0 osd=5 3 2024-04-29T20:23:00.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 15 client.4247.objecter _session_command_op_assign 5 2 2024-04-29T20:23:00.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 4 2024-04-29T20:23:00.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 10 client.4247.objecter _send_command 2 2024-04-29T20:23:00.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 --> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb7e80738d0 con 0x7fb7cc0746c0 2024-04-29T20:23:00.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 10 librados: watch_flush enter 2024-04-29T20:23:00.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 10 librados: watch_flush exit 2024-04-29T20:23:00.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 10 client.4327.objecter close_session for osd.4 2024-04-29T20:23:00.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7f371c074710 msgr2=0x7f371c076b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7f371c074710 0x7f371c076b30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 20 client.4327.objecter put_session s=0x7f371c074040 osd=4 1 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 20 client.4327.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 10 client.4327.objecter successfully canceled tick 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f371c06e2d0 msgr2=0x7f371c070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f371c06e2d0 0x7f371c070790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f372c006fd0 tx=0x7f372c008040 comp rx=0 tx=0).stop 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 10 monclient: shutdown 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3730032c60 msgr2=0x7f37300c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373d6e8640 1 --2- 172.21.15.64:0/3896767166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3730032c60 0x7f37300c7f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3724009960 tx=0x7f37240046c0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.600+0000 7f373cee7640 1 -- 172.21.15.64:0/3896767166 reap_dead start 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.601+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 shutdown_connections 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.601+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 >> 172.21.15.64:0/3896767166 conn(0x7f3730006d20 msgr2=0x7f3730034130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.601+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 shutdown_connections 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.601+0000 7f373d6e8640 1 -- 172.21.15.64:0/3896767166 wait complete. 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.601+0000 7f373d6e8640 1 librados: shutdown 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f895c01c070 con 0x7f89640b5c60 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ddffb640 1 -- 172.21.15.64:0/2627217581 <== osd.5 v2:172.21.15.167:6802/3447508053 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb7e80738d0 con 0x7fb7cc0746c0 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ddffb640 10 client.4247.objecter ms_dispatch 0x7fb7e8000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ddffb640 10 client.4247.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ddffb640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 4 2024-04-29T20:23:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ddffb640 15 client.4247.objecter _session_command_op_remove 5 2 2024-04-29T20:23:00.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7f895bfff640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f894406e2d0 0x7f8944070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7f8962d9d640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f894406e2d0 0x7f8944070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7f8962d9d640 10 monclient: get_auth_request con 0x7f894406e2d0 auth_method 0 2024-04-29T20:23:00.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 10 librados: watch_flush enter 2024-04-29T20:23:00.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 10 librados: watch_flush exit 2024-04-29T20:23:00.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 10 client.4247.objecter close_session for osd.5 2024-04-29T20:23:00.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7fb7cc0746c0 msgr2=0x7fb7cc076ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7fb7cc0746c0 0x7fb7cc076ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 20 client.4247.objecter put_session s=0x7fb7cc073ff0 osd=5 1 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 20 client.4247.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.609+0000 7fb7ef957640 10 client.4247.objecter successfully canceled tick 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb7cc06e280 msgr2=0x7fb7cc070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb7cc06e280 0x7fb7cc070740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7e81412e0 tx=0x7fb7e4009040 comp rx=0 tx=0).stop 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 10 monclient: shutdown 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 msgr2=0x7fb7e8147a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 1 --2- 172.21.15.64:0/2627217581 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb7e80aadd0 0x7fb7e8147a90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80b1570 tx=0x7fb7e0004a40 comp rx=0 tx=0).stop 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7edef4640 1 -- 172.21.15.64:0/2627217581 reap_dead start 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 shutdown_connections 2024-04-29T20:23:00.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 >> 172.21.15.64:0/2627217581 conn(0x7fb7e80b4f80 msgr2=0x7fb7e80ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7f895bfff640 10 client.4336.objecter ms_handle_connect 0x7f894406e2d0 2024-04-29T20:23:00.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.610+0000 7f8962d9d640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f894406e2d0 0x7f8944070790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8954036000 tx=0x7f8954030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.611+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 shutdown_connections 2024-04-29T20:23:00.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.611+0000 7fb7ef957640 1 -- 172.21.15.64:0/2627217581 wait complete. 2024-04-29T20:23:00.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.611+0000 7fb7ef957640 1 librados: shutdown 2024-04-29T20:23:00.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f895c057d30 con 0x7f89640b5c60 2024-04-29T20:23:00.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 10 client.4336.objecter ms_dispatch 0x7f8964000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:00.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 3 client.4336.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 3 client.4336.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 10 client.4336.objecter checking command 1 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f8944074710 0x7f8944076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 20 client.4336.objecter _get_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 15 client.4336.objecter _session_command_op_remove -1 1 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 20 client.4336.objecter _get_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 20 client.4336.objecter get_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 15 client.4336.objecter _session_command_op_assign 7 1 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 4 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 10 client.4336.objecter _send_command 1 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 --> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f895c018ec0 con 0x7f8944074710 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 20 client.4336.objecter dump_active .. 0 homeless 2024-04-29T20:23:00.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f895c055f30 con 0x7f89640b5c60 2024-04-29T20:23:00.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.613+0000 7f895bfff640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:23:00.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.614+0000 7f895a59c640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f8944074710 0x7f8944076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.636 INFO:teuthology.orchestra.run.smithi064.stdout:64424509448 2024-04-29T20:23:00.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-29T20:23:00.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.614+0000 7f895a59c640 10 monclient: get_auth_request con 0x7f8944074710 auth_method 0 2024-04-29T20:23:00.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.618+0000 7f895bfff640 10 client.4336.objecter ms_handle_connect 0x7f8944074710 2024-04-29T20:23:00.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.618+0000 7f895a59c640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f8944074710 0x7f8944076b30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.638 INFO:teuthology.orchestra.run.smithi064.stdout:64424509448 2024-04-29T20:23:00.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-29T20:23:00.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.634+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== osd.0 v2:172.21.15.64:6808/2796311601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f811004be50 con 0x7f81080746c0 2024-04-29T20:23:00.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.634+0000 7f8103fff640 10 client.4321.objecter ms_dispatch 0x7f811c000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.634+0000 7f8103fff640 10 client.4321.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.634+0000 7f8103fff640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 4 2024-04-29T20:23:00.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.634+0000 7f8103fff640 15 client.4321.objecter _session_command_op_remove 0 1 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 15 client.4321.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 20 client.4321.objecter _get_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 20 client.4321.objecter _get_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 15 client.4321.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 20 client.4321.objecter get_session s=0x7f8108073ff0 osd=0 3 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 15 client.4321.objecter _session_command_op_assign 0 2 2024-04-29T20:23:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 4 2024-04-29T20:23:00.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 10 client.4321.objecter _send_command 2 2024-04-29T20:23:00.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.648+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f811c1491c0 con 0x7f81080746c0 2024-04-29T20:23:00.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.652+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== osd.1 v2:172.21.15.64:6809/3421706551 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28e40771e0 con 0x7f28e4074710 2024-04-29T20:23:00.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.652+0000 7f28dffff640 10 client.4333.objecter ms_dispatch 0x7f28f8000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.652+0000 7f28dffff640 10 client.4333.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.652+0000 7f28dffff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 4 2024-04-29T20:23:00.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.652+0000 7f28dffff640 15 client.4333.objecter _session_command_op_remove 1 1 2024-04-29T20:23:00.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.661+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== osd.6 v2:172.21.15.167:6818/3916781155 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3edc0771e0 con 0x7f3edc074710 2024-04-29T20:23:00.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.661+0000 7f3eeaffd640 10 client.4330.objecter ms_dispatch 0x7f3efc000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.661+0000 7f3eeaffd640 10 client.4330.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.661+0000 7f3eeaffd640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 4 2024-04-29T20:23:00.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.661+0000 7f3eeaffd640 15 client.4330.objecter _session_command_op_remove 6 1 2024-04-29T20:23:00.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.664+0000 7f8103fff640 1 -- 172.21.15.64:0/3599402015 <== osd.0 v2:172.21.15.64:6808/2796311601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f811c1491c0 con 0x7f81080746c0 2024-04-29T20:23:00.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.664+0000 7f8103fff640 10 client.4321.objecter ms_dispatch 0x7f811c000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.664+0000 7f8103fff640 10 client.4321.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.664+0000 7f8103fff640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 4 2024-04-29T20:23:00.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.664+0000 7f8103fff640 15 client.4321.objecter _session_command_op_remove 0 2 2024-04-29T20:23:00.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.666+0000 7f81218e4640 10 librados: watch_flush enter 2024-04-29T20:23:00.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.666+0000 7f81218e4640 10 librados: watch_flush exit 2024-04-29T20:23:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.666+0000 7f81218e4640 10 client.4321.objecter close_session for osd.0 2024-04-29T20:23:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.666+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f81080746c0 msgr2=0x7f8108076ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.666+0000 7f81218e4640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f81080746c0 0x7f8108076ae0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.667+0000 7f81218e4640 20 client.4321.objecter put_session s=0x7f8108073ff0 osd=0 1 2024-04-29T20:23:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.667+0000 7f81218e4640 20 client.4321.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.667+0000 7f81218e4640 10 client.4321.objecter successfully canceled tick 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.668+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f810806e280 msgr2=0x7f8108070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f810806e280 0x7f8108070740 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81140070b0 tx=0x7f8114007040 comp rx=0 tx=0).stop 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 10 monclient: shutdown 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f811c150200 msgr2=0x7f811c13c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 --2- 172.21.15.64:0/3599402015 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f811c150200 0x7f811c13c440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8110034040 tx=0x7f81100046c0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f811b7fe640 1 -- 172.21.15.64:0/3599402015 reap_dead start 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 shutdown_connections 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 >> 172.21.15.64:0/3599402015 conn(0x7f811c06c960 msgr2=0x7f811c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 shutdown_connections 2024-04-29T20:23:00.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.669+0000 7f81218e4640 1 -- 172.21.15.64:0/3599402015 wait complete. 2024-04-29T20:23:00.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.672+0000 7f81218e4640 1 librados: shutdown 2024-04-29T20:23:00.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.677+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== osd.3 v2:172.21.15.64:6824/471606907 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9468031710 con 0x7f945c0746c0 2024-04-29T20:23:00.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.677+0000 7f946dffb640 10 client.4305.objecter ms_dispatch 0x7f9478000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.677+0000 7f946dffb640 10 client.4305.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.677+0000 7f946dffb640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 4 2024-04-29T20:23:00.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.677+0000 7f946dffb640 15 client.4305.objecter _session_command_op_remove 3 1 2024-04-29T20:23:00.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.680+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== osd.7 v2:172.21.15.167:6810/2258625350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f895c018ec0 con 0x7f8944074710 2024-04-29T20:23:00.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.680+0000 7f895bfff640 10 client.4336.objecter ms_dispatch 0x7f8964000b90 command_reply(tid 1: 0 ) 2024-04-29T20:23:00.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.680+0000 7f895bfff640 10 client.4336.objecter _finish_command 1 = system:0 2024-04-29T20:23:00.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.680+0000 7f895bfff640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 4 2024-04-29T20:23:00.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.680+0000 7f895bfff640 15 client.4336.objecter _session_command_op_remove 7 1 2024-04-29T20:23:00.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 15 client.4333.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 20 client.4333.objecter _get_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 20 client.4333.objecter _get_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 15 client.4333.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 20 client.4333.objecter get_session s=0x7f28e4074040 osd=1 3 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 15 client.4333.objecter _session_command_op_assign 1 2 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 4 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 10 client.4333.objecter _send_command 2 2024-04-29T20:23:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.682+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 --> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f28f80712a0 con 0x7f28e4074710 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stdout:94489280521 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 15 client.4330.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 20 client.4330.objecter _get_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 20 client.4330.objecter _get_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 15 client.4330.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 20 client.4330.objecter get_session s=0x7f3edc074040 osd=6 3 2024-04-29T20:23:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 15 client.4330.objecter _session_command_op_assign 6 2 2024-04-29T20:23:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 4 2024-04-29T20:23:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 10 client.4330.objecter _send_command 2 2024-04-29T20:23:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.692+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 --> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3efc1dc820 con 0x7f3edc074710 2024-04-29T20:23:00.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-29T20:23:00.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28dffff640 1 -- 172.21.15.64:0/1228931810 <== osd.1 v2:172.21.15.64:6809/3421706551 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f28f80712a0 con 0x7f28e4074710 2024-04-29T20:23:00.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28dffff640 10 client.4333.objecter ms_dispatch 0x7f28f8000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28dffff640 10 client.4333.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28dffff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 4 2024-04-29T20:23:00.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28dffff640 15 client.4333.objecter _session_command_op_remove 1 2 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 10 librados: watch_flush enter 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 10 librados: watch_flush exit 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 10 client.4333.objecter close_session for osd.1 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7f28e4074710 msgr2=0x7f28e4076b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7f28e4074710 0x7f28e4076b30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 20 client.4333.objecter put_session s=0x7f28e4074040 osd=1 1 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 20 client.4333.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.702+0000 7f28f7fff640 10 client.4333.objecter successfully canceled tick 2024-04-29T20:23:00.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f28e406e2d0 msgr2=0x7f28e4070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f28e406e2d0 0x7f28e4070790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006a90 tx=0x7f28ec006a20 comp rx=0 tx=0).stop 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 10 monclient: shutdown 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28f80aab10 msgr2=0x7f28f80b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 1 --2- 172.21.15.64:0/1228931810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28f80aab10 0x7f28f80b39b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28e800e7d0 tx=0x7f28e800ec90 comp rx=0 tx=0).stop 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f77fe640 1 -- 172.21.15.64:0/1228931810 reap_dead start 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 shutdown_connections 2024-04-29T20:23:00.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.703+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 >> 172.21.15.64:0/1228931810 conn(0x7f28f8068320 msgr2=0x7f28f8142400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.704+0000 7f3eeaffd640 1 -- 172.21.15.64:0/1017661381 <== osd.6 v2:172.21.15.167:6818/3916781155 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3efc1dc820 con 0x7f3edc074710 2024-04-29T20:23:00.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.704+0000 7f3eeaffd640 10 client.4330.objecter ms_dispatch 0x7f3efc000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.704+0000 7f3eeaffd640 10 client.4330.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.704+0000 7f3eeaffd640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 4 2024-04-29T20:23:00.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.704+0000 7f3eeaffd640 15 client.4330.objecter _session_command_op_remove 6 2 2024-04-29T20:23:00.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 10 librados: watch_flush enter 2024-04-29T20:23:00.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 10 librados: watch_flush exit 2024-04-29T20:23:00.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 10 client.4330.objecter close_session for osd.6 2024-04-29T20:23:00.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 shutdown_connections 2024-04-29T20:23:00.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f28f7fff640 1 -- 172.21.15.64:0/1228931810 wait complete. 2024-04-29T20:23:00.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.707+0000 7f28f7fff640 1 librados: shutdown 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7f3edc074710 msgr2=0x7f3edc076b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7f3edc074710 0x7f3edc076b30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 20 client.4330.objecter put_session s=0x7f3edc074040 osd=6 1 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 20 client.4330.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 10 client.4330.objecter successfully canceled tick 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3edc06e2d0 msgr2=0x7f3edc070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3edc06e2d0 0x7f3edc070790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009710 tx=0x7f3eec006c90 comp rx=0 tx=0).stop 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 10 monclient: shutdown 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.705+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc0abc10 msgr2=0x7f3efc0b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3f00af3640 1 --2- 172.21.15.64:0/1017661381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3efc0abc10 0x7f3efc0b4510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c440 tx=0x7f3ef000c900 comp rx=0 tx=0).stop 2024-04-29T20:23:00.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3efad9d640 1 -- 172.21.15.64:0/1017661381 reap_dead start 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 shutdown_connections 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 >> 172.21.15.64:0/1017661381 conn(0x7f3efc0b4b60 msgr2=0x7f3efc0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 shutdown_connections 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3f00af3640 1 -- 172.21.15.64:0/1017661381 wait complete. 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.706+0000 7f3f00af3640 1 librados: shutdown 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 10 client.4305.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 15 client.4305.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 20 client.4305.objecter _get_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 20 client.4305.objecter _get_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 15 client.4305.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 20 client.4305.objecter get_session s=0x7f945c073ff0 osd=3 3 2024-04-29T20:23:00.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 15 client.4305.objecter _session_command_op_assign 3 2 2024-04-29T20:23:00.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 4 2024-04-29T20:23:00.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 10 client.4305.objecter _send_command 2 2024-04-29T20:23:00.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.708+0000 7f947fa09640 1 -- 172.21.15.64:0/1665183110 --> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f947813f230 con 0x7f945c0746c0 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 15 client.4336.objecter _session_command_op_assign -1 2 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 20 client.4336.objecter _get_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 20 client.4336.objecter _get_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 15 client.4336.objecter _session_command_op_remove -1 2 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 20 client.4336.objecter get_session s=0x7f8944074040 osd=7 3 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 15 client.4336.objecter _session_command_op_assign 7 2 2024-04-29T20:23:00.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 4 2024-04-29T20:23:00.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 10 client.4336.objecter _send_command 2 2024-04-29T20:23:00.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.722+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 --> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89640b53f0 con 0x7f8944074710 2024-04-29T20:23:00.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f946dffb640 1 -- 172.21.15.64:0/1665183110 <== osd.3 v2:172.21.15.64:6824/471606907 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f947813f230 con 0x7f945c0746c0 2024-04-29T20:23:00.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f946dffb640 10 client.4305.objecter ms_dispatch 0x7f9478000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f946dffb640 10 client.4305.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f946dffb640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 4 2024-04-29T20:23:00.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f946dffb640 15 client.4305.objecter _session_command_op_remove 3 2 2024-04-29T20:23:00.730 INFO:teuthology.orchestra.run.smithi064.stdout:94489280522 2024-04-29T20:23:00.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 10 librados: watch_flush enter 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 10 librados: watch_flush exit 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 10 client.4305.objecter close_session for osd.3 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f945c0746c0 msgr2=0x7f945c076ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f945c0746c0 0x7f945c076ae0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 20 client.4305.objecter put_session s=0x7f945c073ff0 osd=3 1 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 20 client.4305.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 10 client.4305.objecter successfully canceled tick 2024-04-29T20:23:00.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f945c06e280 msgr2=0x7f945c070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.728+0000 7f94637fe640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f945c06e280 0x7f945c070740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f94780b4b10 tx=0x7f9470009040 comp rx=0 tx=0).stop 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.729+0000 7f94637fe640 10 monclient: shutdown 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.729+0000 7f94637fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.729+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94780ab1a0 msgr2=0x7f94780b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.729+0000 7f94637fe640 1 --2- 172.21.15.64:0/1665183110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94780ab1a0 0x7f94780b44c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f946802b7f0 tx=0x7f94680046c0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.729+0000 7f947dfa6640 1 -- 172.21.15.64:0/1665183110 reap_dead start 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.731+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 shutdown_connections 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.732+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 >> 172.21.15.64:0/1665183110 conn(0x7f94780b4b60 msgr2=0x7f947813fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.732+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 shutdown_connections 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.732+0000 7f94637fe640 1 -- 172.21.15.64:0/1665183110 wait complete. 2024-04-29T20:23:00.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.732+0000 7f94637fe640 1 librados: shutdown 2024-04-29T20:23:00.741 INFO:teuthology.orchestra.run.smithi064.stdout:85899345929 2024-04-29T20:23:00.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-29T20:23:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.740+0000 7f895bfff640 1 -- 172.21.15.64:0/704879801 <== osd.7 v2:172.21.15.167:6810/2258625350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f89640b53f0 con 0x7f8944074710 2024-04-29T20:23:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.740+0000 7f895bfff640 10 client.4336.objecter ms_dispatch 0x7f8964000b90 command_reply(tid 2: 0 ) 2024-04-29T20:23:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.740+0000 7f895bfff640 10 client.4336.objecter _finish_command 2 = system:0 2024-04-29T20:23:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.740+0000 7f895bfff640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 4 2024-04-29T20:23:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.740+0000 7f895bfff640 15 client.4336.objecter _session_command_op_remove 7 2 2024-04-29T20:23:00.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 10 librados: watch_flush enter 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 10 librados: watch_flush exit 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 10 client.4336.objecter close_session for osd.7 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f8944074710 msgr2=0x7f8944076b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f8944074710 0x7f8944076b30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 20 client.4336.objecter put_session s=0x7f8944074040 osd=7 1 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 20 client.4336.objecter shutdown clearing up homeless session... 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 10 client.4336.objecter successfully canceled tick 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f894406e2d0 msgr2=0x7f8944070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f894406e2d0 0x7f8944070790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8954036000 tx=0x7f8954030000 comp rx=0 tx=0).stop 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 10 monclient: shutdown 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b5c60 msgr2=0x7f89641d8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 --2- 172.21.15.64:0/704879801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89640b5c60 0x7f89641d8950 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f895c00b970 tx=0x7f895c00be30 comp rx=0 tx=0).stop 2024-04-29T20:23:00.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f896359e640 1 -- 172.21.15.64:0/704879801 reap_dead start 2024-04-29T20:23:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 shutdown_connections 2024-04-29T20:23:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.741+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 >> 172.21.15.64:0/704879801 conn(0x7f896406c960 msgr2=0x7f89641403f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.742+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 shutdown_connections 2024-04-29T20:23:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.742+0000 7f8969030640 1 -- 172.21.15.64:0/704879801 wait complete. 2024-04-29T20:23:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.742+0000 7f8969030640 1 librados: shutdown 2024-04-29T20:23:00.756 INFO:teuthology.orchestra.run.smithi064.stdout:94489280522 2024-04-29T20:23:00.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-29T20:23:00.764 INFO:teuthology.orchestra.run.smithi064.stdout:64424509446 2024-04-29T20:23:00.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-29T20:23:00.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.821+0000 7f364c10d640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.821+0000 7f364c10d640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.821+0000 7f364c10d640 10 monclient: monmap: 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.822192+0000 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.822192+0000 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.823 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.822+0000 7f364c10d640 1 Processor -- start 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.822+0000 7f364c10d640 1 -- start start 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.822+0000 7f364c10d640 10 monclient: init 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f36440b2ee0 0x7f3644144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient(hunting): picked mon.noname-c con 0x7f36440b2ee0 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36440b37b0 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3644150200 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364406ef20 con 0x7f36440b37b0 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364406f200 con 0x7f3644150200 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f36440b32c0 con 0x7f36440b2ee0 2024-04-29T20:23:00.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.823+0000 7f364c10d640 10 monclient(hunting): authenticate will time out at 1529.983234s 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f3649ea9640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f36440b2ee0 0x7f3644144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f36496a8640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f36440b37b0 0x7f36440b3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51862/0 (socket says 172.21.15.64:51862) 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f36496a8640 1 -- 172.21.15.64:0/1094244182 learned_addr learned my addr 172.21.15.64:0/1094244182 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f3649ea9640 10 monclient(hunting): get_auth_request con 0x7f36440b2ee0 auth_method 0 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f3649ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f3649ea9640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.824+0000 7f3649ea9640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.826+0000 7f2ffecce640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.826+0000 7f2ffecce640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.826+0000 7f2ffecce640 10 monclient: monmap: 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.827205+0000 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.827205+0000 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.828 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.827+0000 7f3649ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.827+0000 7f3649ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.827+0000 7f3649ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.827+0000 7f2ffecce640 1 Processor -- start 2024-04-29T20:23:00.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.827+0000 7f2ffecce640 1 -- start start 2024-04-29T20:23:00.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.827+0000 7f2ffecce640 10 monclient: init 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ff80b8b90 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ff80b8f70 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2ff80aa940 0x7f2ff80b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff80aa940 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.828+0000 7f2ffecce640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff80ac440 con 0x7f2ff80b8b90 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.829+0000 7f2ffecce640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff80b3040 con 0x7f2ff80b8f70 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.829+0000 7f2ffecce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.829+0000 7f2ffecce640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff81470a0 con 0x7f2ff80aa940 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.829+0000 7f2ffecce640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.829+0000 7f2ffecce640 10 monclient(hunting): authenticate will time out at 1529.988580s 2024-04-29T20:23:00.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3648ea7640 1 -- 172.21.15.64:0/1094244182 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219494324 0 0) 0x7f364406ef20 con 0x7f36440b37b0 2024-04-29T20:23:00.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3648ea7640 10 monclient(hunting): _negotiate 2024-04-29T20:23:00.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3648ea7640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3648ea7640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3648ea7640 10 monclient(hunting): my global_id is 4314 2024-04-29T20:23:00.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3648ea7640 1 -- 172.21.15.64:0/1094244182 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3630003520 con 0x7f36440b37b0 2024-04-29T20:23:00.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.831+0000 7f2ffca6a640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2ff80b8b90 0x7f2ff8140a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51870/0 (socket says 172.21.15.64:51870) 2024-04-29T20:23:00.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.831+0000 7f2ffca6a640 1 -- 172.21.15.64:0/2772526799 learned_addr learned my addr 172.21.15.64:0/2772526799 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-04-29T20:23:00.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 1 -- 172.21.15.64:0/1094244182 >> v1:172.21.15.167:6789/0 conn(0x7f3644150200 legacy=0x7f36441525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 1 -- 172.21.15.64:0/1094244182 >> v1:172.21.15.64:6789/0 conn(0x7f36440b37b0 legacy=0x7f36440b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.830+0000 7f3649ea9640 1 -- 172.21.15.64:0/1094244182 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3644148210 con 0x7f36440b2ee0 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.833+0000 7f3649ea9640 1 --2- 172.21.15.64:0/1094244182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36440b2ee0 0x7f3644144bd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c02bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632cce8e18a80d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.834+0000 7f2ffd26b640 1 --2- 172.21.15.64:0/2772526799 >> v2:172.21.15.167:3300/0 conn(0x7f2ff80aa940 0x7f2ff80b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.835+0000 7f2ffd26b640 10 monclient(hunting): get_auth_request con 0x7f2ff80aa940 auth_method 0 2024-04-29T20:23:00.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.835+0000 7f2ffd26b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.835+0000 7f2ffd26b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.835+0000 7f2ffd26b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.837+0000 7f2ffd26b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.837+0000 7f2ffd26b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.837+0000 7f2ffd26b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.839+0000 7f46eed9a640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.839+0000 7f46eed9a640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.839+0000 7f46eed9a640 10 monclient: monmap: 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.840521+0000 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.840521+0000 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.841 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 1 -- 172.21.15.64:0/2772526799 >> v1:172.21.15.167:6789/0 conn(0x7f2ff80b8f70 legacy=0x7f2ff8146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 1 -- 172.21.15.64:0/2772526799 >> v1:172.21.15.64:6789/0 conn(0x7f2ff80b8b90 legacy=0x7f2ff8140a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.840+0000 7f2ffd26b640 1 -- 172.21.15.64:0/2772526799 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff8150d50 con 0x7f2ff80aa940 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.841+0000 7f46eed9a640 1 Processor -- start 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.841+0000 7f46eed9a640 1 -- start start 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.841+0000 7f46eed9a640 10 monclient: init 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46e80b2ee0 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f46e80b37b0 0x7f46e80b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): picked mon.noname-a con 0x7f46e80b37b0 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e8150200 0x7f46e81525f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): picked mon.noname-b con 0x7f46e8150200 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e81452e0 con 0x7f46e80b2ee0 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f46e80b32c0 con 0x7f46e80b37b0 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f46e806ecd0 con 0x7f46e8150200 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.842+0000 7f46eed9a640 10 monclient(hunting): authenticate will time out at 1530.001919s 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.843+0000 7f46ed337640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e8150200 0x7f46e81525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46ed337640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e8150200 0x7f46e81525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36724/0 (socket says 172.21.15.64:36724) 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46ed337640 1 -- 172.21.15.64:0/2659794544 learned_addr learned my addr 172.21.15.64:0/2659794544 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46ed337640 10 monclient(hunting): get_auth_request con 0x7f46e8150200 auth_method 0 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46ed337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46ed337640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46ed337640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f46e7fff640 1 --2- 172.21.15.64:0/2659794544 >> v2:172.21.15.64:3300/0 conn(0x7f46e80b37b0 0x7f46e80b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 1 -- 172.21.15.64:0/1094244182 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c039070 con 0x7f36440b2ee0 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: dump: 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.848 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 1 -- 172.21.15.64:0/1094244182 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f363c02ccb0 con 0x7f36440b2ee0 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f364c10d640 5 monclient: authenticate success, global_id 4339 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f364c10d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f364c10d640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 1 -- 172.21.15.64:0/1094244182 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c034660 con 0x7f36440b2ee0 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f364c10d640 4 set_mon_vals no callback set 2024-04-29T20:23:00.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.844+0000 7f3648ea7640 10 monclient: dump: 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f364c10d640 10 monclient: shutdown 2024-04-29T20:23:00.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f364c10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f364c10d640 1 -- 172.21.15.64:0/1094244182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36440b2ee0 msgr2=0x7f3644144bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f364c10d640 1 --2- 172.21.15.64:0/1094244182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36440b2ee0 0x7f3644144bd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c02bf00 comp rx=0 tx=0).stop 2024-04-29T20:23:00.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f46ed337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f46ed337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f46ed337640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.845+0000 7f2ffd26b640 1 --2- 172.21.15.64:0/2772526799 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80aa940 0x7f2ff80b2b00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0097a0 tx=0x7f2fec02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56c269d4105b72df server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.846+0000 7f46e7fff640 10 monclient(hunting): get_auth_request con 0x7f46e80b37b0 auth_method 0 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.846+0000 7f46e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.846+0000 7f46e7fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.846+0000 7f46e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.847+0000 7f46e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.847+0000 7f46e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.847+0000 7f46e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.849+0000 7f46e7fff640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-04-29T20:23:00.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.849+0000 7f46e7fff640 1 -- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e8150200 msgr2=0x7f46e81525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 1 --2- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e8150200 0x7f46e81525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 1 -- 172.21.15.64:0/2659794544 >> v1:172.21.15.64:6789/0 conn(0x7f46e80b2ee0 legacy=0x7f46e8144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.850+0000 7f46e7fff640 1 -- 172.21.15.64:0/2659794544 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e806ef20 con 0x7f46e80b37b0 2024-04-29T20:23:00.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.852+0000 7f364c10d640 1 -- 172.21.15.64:0/1094244182 shutdown_connections 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.852+0000 7f364c10d640 1 --2- 172.21.15.64:0/1094244182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36440b2ee0 0x7f3644144bd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.852+0000 7f364c10d640 1 -- 172.21.15.64:0/1094244182 >> 172.21.15.64:0/1094244182 conn(0x7f364406c960 msgr2=0x7f364406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.853+0000 7f364c10d640 1 -- 172.21.15.64:0/1094244182 shutdown_connections 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.853+0000 7f364c10d640 1 -- 172.21.15.64:0/1094244182 wait complete. 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.854+0000 7f364c10d640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.854+0000 7f364c10d640 10 monclient: monmap: 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.855 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.854+0000 7f364c10d640 1 librados: starting msgr at 2024-04-29T20:23:00.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.854+0000 7f364c10d640 1 librados: starting objecter 2024-04-29T20:23:00.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 1 Processor -- start 2024-04-29T20:23:00.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 1 -- start start 2024-04-29T20:23:00.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 1 librados: setting wanted keys 2024-04-29T20:23:00.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 1 librados: calling monclient init 2024-04-29T20:23:00.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 10 monclient: init 2024-04-29T20:23:00.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f46e7fff640 1 --2- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e80b3c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3841ff4d08b8f043 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.855+0000 7f364c10d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 0x7f3644150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 10 monclient(hunting): picked mon.noname-c con 0x7f36440b2ee0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36440b37b0 0x7f36441351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 10 monclient(hunting): picked mon.noname-a con 0x7f36440b37b0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3644150200 0x7f364412f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 10 monclient(hunting): picked mon.noname-b con 0x7f3644150200 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.856+0000 7f364c10d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3644148730 con 0x7f36440b37b0 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f364406ecd0 con 0x7f3644150200 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 0x7f3644150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f3644147de0 con 0x7f36440b2ee0 2024-04-29T20:23:00.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f364c10d640 10 monclient(hunting): authenticate will time out at 1530.017096s 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 0x7f3644150850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36728/0 (socket says 172.21.15.64:36728) 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 1 -- 172.21.15.64:0/2981763944 learned_addr learned my addr 172.21.15.64:0/2981763944 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 10 monclient(hunting): get_auth_request con 0x7f36440b2ee0 auth_method 0 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.857+0000 7f3649ea9640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.859+0000 7f3649ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.859+0000 7f3649ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.859+0000 7f3649ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 1 -- 172.21.15.64:0/2772526799 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0315a0 con 0x7f2ff80aa940 2024-04-29T20:23:00.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 10 monclient: dump: 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f2ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f46e77fe640 1 -- 172.21.15.64:0/2659794544 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc039070 con 0x7f46e80b37b0 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f46e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.862+0000 7f46e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46e77fe640 10 monclient: dump: 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.866 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46e77fe640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46e77fe640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46eed9a640 5 monclient: authenticate success, global_id 4320 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46eed9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46e77fe640 1 -- 172.21.15.64:0/2659794544 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc0316f0 con 0x7f46e80b37b0 2024-04-29T20:23:00.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46e77fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f36496a8640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36440b37b0 0x7f36441351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f36496a8640 10 monclient(hunting): get_auth_request con 0x7f36440b37b0 auth_method 0 2024-04-29T20:23:00.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f36496a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f36496a8640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f36496a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f364a6aa640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3644150200 0x7f364412f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46eed9a640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.863+0000 7f46eed9a640 4 set_mon_vals no callback set 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f46eed9a640 10 monclient: shutdown 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f46eed9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f46eed9a640 1 -- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 msgr2=0x7f46e80b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.864+0000 7f46eed9a640 1 --2- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e80b3c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b790 comp rx=0 tx=0).stop 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f36496a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f36496a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f36496a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f46eed9a640 1 -- 172.21.15.64:0/2659794544 shutdown_connections 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f46eed9a640 1 --2- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e8150200 0x7f46e81525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f46eed9a640 1 --2- 172.21.15.64:0/2659794544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e80b3c20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f46eed9a640 1 -- 172.21.15.64:0/2659794544 >> 172.21.15.64:0/2659794544 conn(0x7f46e806c960 msgr2=0x7f46e806cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.865+0000 7f46eed9a640 1 -- 172.21.15.64:0/2659794544 shutdown_connections 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.866+0000 7f46eed9a640 1 -- 172.21.15.64:0/2659794544 wait complete. 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.866+0000 7f46eed9a640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.866+0000 7f46eed9a640 10 monclient: monmap: 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.871 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 1 librados: starting msgr at 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 1 librados: starting objecter 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 1 Processor -- start 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 1 -- start start 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 1 librados: setting wanted keys 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 1 librados: calling monclient init 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.867+0000 7f46eed9a640 10 monclient: init 2024-04-29T20:23:00.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 1 -- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3644150200 msgr2=0x7f364412f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3644150200 0x7f364412f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 1 -- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36440b37b0 msgr2=0x7f36441351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36440b37b0 0x7f36441351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 1 monclient: found mon.noname-c 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f3649ea9640 1 -- 172.21.15.64:0/2981763944 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3644151cb0 con 0x7f36440b2ee0 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ffecce640 5 monclient: authenticate success, global_id 4345 2024-04-29T20:23:00.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ffecce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ff77fe640 1 -- 172.21.15.64:0/2772526799 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fec037070 con 0x7f2ff80aa940 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ff77fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ff77fe640 1 -- 172.21.15.64:0/2772526799 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec031960 con 0x7f2ff80aa940 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ff77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.868+0000 7f2ff77fe640 10 monclient: dump: 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.874 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.875 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.875 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.869+0000 7f46eed9a640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.869+0000 7f46eed9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.869+0000 7f46eed9a640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e80b2ee0 0x7f46e80b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.869+0000 7f46eed9a640 10 monclient(hunting): picked mon.noname-c con 0x7f46e80b2ee0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.869+0000 7f46eed9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e8148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 10 monclient(hunting): picked mon.noname-a con 0x7f46e80b37b0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 0x7f46e81e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 10 monclient(hunting): picked mon.noname-b con 0x7f46e8150200 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f46e80b4160 con 0x7f46e80b37b0 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46e806da10 con 0x7f46e8150200 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.870+0000 7f46eed9a640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f46e8148730 con 0x7f46e80b2ee0 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46eed9a640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46eed9a640 10 monclient(hunting): authenticate will time out at 1530.030710s 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 0x7f46e81e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 0x7f46e81e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33602/0 (socket says 172.21.15.64:33602) 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 1 -- 172.21.15.64:0/1487215457 learned_addr learned my addr 172.21.15.64:0/1487215457 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 10 monclient(hunting): get_auth_request con 0x7f46e8150200 auth_method 0 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f46ed337640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.871+0000 7f2ffecce640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.872+0000 7f2ffecce640 4 set_mon_vals no callback set 2024-04-29T20:23:00.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.872+0000 7f2ffecce640 10 monclient: shutdown 2024-04-29T20:23:00.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.872+0000 7f2ffecce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.872+0000 7f2ffecce640 1 -- 172.21.15.64:0/2772526799 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80aa940 msgr2=0x7f2ff80b2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.872+0000 7f2ffecce640 1 --2- 172.21.15.64:0/2772526799 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80aa940 0x7f2ff80b2b00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0097a0 tx=0x7f2fec02b840 comp rx=0 tx=0).stop 2024-04-29T20:23:00.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ed337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ed337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ed337640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f2ffecce640 1 -- 172.21.15.64:0/2772526799 shutdown_connections 2024-04-29T20:23:00.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f2ffecce640 1 --2- 172.21.15.64:0/2772526799 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80aa940 0x7f2ff80b2b00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f2ffecce640 1 -- 172.21.15.64:0/2772526799 >> 172.21.15.64:0/2772526799 conn(0x7f2ff8067720 msgr2=0x7f2ff8067b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ecb36640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e80b2ee0 0x7f46e80b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ecb36640 10 monclient(hunting): get_auth_request con 0x7f46e80b2ee0 auth_method 0 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ecb36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ecb36640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.873+0000 7f46ecb36640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46e7fff640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e8148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46e7fff640 10 monclient(hunting): get_auth_request con 0x7f46e80b37b0 auth_method 0 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46e7fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46ecb36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46ecb36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.874+0000 7f46ecb36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.875+0000 7f2ffecce640 1 -- 172.21.15.64:0/2772526799 shutdown_connections 2024-04-29T20:23:00.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.875+0000 7f2ffecce640 1 -- 172.21.15.64:0/2772526799 wait complete. 2024-04-29T20:23:00.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.876+0000 7f2ffecce640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.876+0000 7f2ffecce640 10 monclient: monmap: 2024-04-29T20:23:00.886 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.876+0000 7f2ffecce640 1 librados: starting msgr at 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.876+0000 7f2ffecce640 1 librados: starting objecter 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.877+0000 7f2ffecce640 1 Processor -- start 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.877+0000 7f2ffecce640 1 -- start start 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.877+0000 7f2ffecce640 1 librados: setting wanted keys 2024-04-29T20:23:00.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.877+0000 7f2ffecce640 1 librados: calling monclient init 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.877+0000 7f2ffecce640 10 monclient: init 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff80aa940 0x7f2ff80acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff80aa940 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 0x7f2ff80b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff80b8b90 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2ff80b12a0 0x7f2ff81400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff80b12a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff814f0c0 con 0x7f2ff80aa940 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff814ef40 con 0x7f2ff80b8b90 2024-04-29T20:23:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff814f240 con 0x7f2ff80b12a0 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.878+0000 7f2ffecce640 10 monclient(hunting): authenticate will time out at 1530.037834s 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 0x7f2ff80b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 0x7f2ff80b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33612/0 (socket says 172.21.15.64:33612) 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 1 -- 172.21.15.64:0/1169751715 learned_addr learned my addr 172.21.15.64:0/1169751715 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 10 monclient(hunting): get_auth_request con 0x7f2ff80b8b90 auth_method 0 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.879+0000 7f2ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-04-29T20:23:00.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 1 -- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e80b2ee0 msgr2=0x7f46e80b8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e80b2ee0 0x7f46e80b8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 1 -- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 msgr2=0x7f46e8148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e8148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.881+0000 7f2ffca6a640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff80aa940 0x7f2ff80acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.881+0000 7f2ffca6a640 10 monclient(hunting): get_auth_request con 0x7f2ff80aa940 auth_method 0 2024-04-29T20:23:00.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.881+0000 7f2ffca6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.881+0000 7f2ffca6a640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.881+0000 7f2ffca6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.880+0000 7f46ed337640 1 -- 172.21.15.64:0/1487215457 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e814a380 con 0x7f46e8150200 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 1 -- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2ff80b12a0 msgr2=0x7f2ff81400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2ff80b12a0 0x7f2ff81400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 1 -- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff80aa940 msgr2=0x7f2ff80acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff80aa940 0x7f2ff80acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 1 monclient: found mon.noname-b 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f2ff7fff640 1 -- 172.21.15.64:0/1169751715 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff80ae310 con 0x7f2ff80b8b90 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f3649ea9640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 0x7f3644150850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.883+0000 7f363a7fc640 10 client.?.objecter ms_handle_connect 0x7f36440b2ee0 2024-04-29T20:23:00.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.884+0000 7f363a7fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.885+0000 7f46ed337640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 0x7f46e81e5df0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46e000eed0 tx=0x7f46e000b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.885+0000 7f46e57fa640 10 client.?.objecter ms_handle_connect 0x7f46e8150200 2024-04-29T20:23:00.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.885+0000 7f46e57fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.885+0000 7f2ff7fff640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 0x7f2ff80b0d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800c430 tx=0x7f2fe800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.886+0000 7f2ff57fa640 10 client.?.objecter ms_handle_connect 0x7f2ff80b8b90 2024-04-29T20:23:00.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.886+0000 7f2ff57fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.889+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e001a070 con 0x7f46e8150200 2024-04-29T20:23:00.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.889+0000 7f46e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.889+0000 7f46e57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.889+0000 7f46e57fa640 10 monclient: dump: 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 5 monclient: authenticate success, global_id 4348 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 10 monclient: _renew_subs 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e80b9810 con 0x7f46e8150200 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 10 monclient: _renew_subs 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e814a980 con 0x7f46e8150200 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46eed9a640 1 librados: init done 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e000bb10 con 0x7f46e8150200 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0003990 con 0x7f46e8150200 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.890+0000 7f46e57fa640 10 monclient: dump: 2024-04-29T20:23:00.897 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.891+0000 7f46e67fc640 4 set_mon_vals no callback set 2024-04-29T20:23:00.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f46eed9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c044030 con 0x7f36440b2ee0 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: dump: 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.899 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f363c02cab0 con 0x7f36440b2ee0 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c039040 con 0x7f36440b2ee0 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363a7fc640 10 monclient: dump: 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.900 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.892+0000 7f363bfff640 4 set_mon_vals no callback set 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 5 monclient: authenticate success, global_id 4259 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 10 monclient: _renew_subs 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3644151f90 con 0x7f36440b2ee0 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 10 monclient: _renew_subs 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f364414fed0 con 0x7f36440b2ee0 2024-04-29T20:23:00.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f364c10d640 1 librados: init done 2024-04-29T20:23:00.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f46eed9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.894+0000 7f46eed9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.894+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c8005080 con 0x7f46e8150200 2024-04-29T20:23:00.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f87c33f7640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f87c33f7640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.893+0000 7f87c33f7640 10 monclient: monmap: 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.893979+0000 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.893979+0000 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.903 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f364c10d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f364c10d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f364c10d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:00.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f364414fcd0 con 0x7f36440b2ee0 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.894+0000 7f87c33f7640 1 Processor -- start 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.894+0000 7f87c33f7640 1 -- start start 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.894+0000 7f87c33f7640 10 monclient: init 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f87c33f7640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f87c33f7640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f87c33f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87bc0aad50 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f87c33f7640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f87bc14ba90 0x7f87bc0abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.895+0000 7f87c33f7640 10 monclient(hunting): picked mon.noname-c con 0x7f87bc14ba90 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f87bc0ad6a0 0x7f87bc144b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 10 monclient(hunting): picked mon.noname-a con 0x7f87bc0ad6a0 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc0acfa0 con 0x7f87bc0aad50 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc14aef0 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc0ab130 con 0x7f87bc14ba90 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.896+0000 7f87c33f7640 10 monclient(hunting): authenticate will time out at 1530.055643s 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.898+0000 7f3bce598640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.898+0000 7f3bce598640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.898+0000 7f3bce598640 10 monclient: monmap: 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.898898+0000 2024-04-29T20:23:00.907 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.898898+0000 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.899+0000 7f3bce598640 1 Processor -- start 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.899+0000 7f3bce598640 1 -- start start 2024-04-29T20:23:00.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.899+0000 7f3bce598640 10 monclient: init 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8017070 con 0x7f2ff80b8b90 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: dump: 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.909 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe80052d0 con 0x7f2ff80b8b90 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8003830 con 0x7f2ff80b8b90 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ff57fa640 10 monclient: dump: 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.910 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.911 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.911 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.911 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.911 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.911 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.911 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-29T20:23:00.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bc80b2230 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f3bc80aa460 0x7f3bc80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): picked mon.noname-c con 0x7f3bc80aa460 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3bc80b5c60 0x7f3bc814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): picked mon.noname-a con 0x7f3bc80b5c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80ac1f0 con 0x7f3bc80b2230 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc80ac4d0 con 0x7f3bc80b5c60 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc80b3440 con 0x7f3bc80aa460 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f3bce598640 10 monclient(hunting): authenticate will time out at 1530.059922s 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 5 monclient: authenticate success, global_id 4351 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 10 monclient: _renew_subs 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff80ae5f0 con 0x7f2ff80b8b90 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 10 monclient: _renew_subs 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff8071880 con 0x7f2ff80b8b90 2024-04-29T20:23:00.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.900+0000 7f2ffecce640 1 librados: init done 2024-04-29T20:23:00.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.901+0000 7f2ff67fc640 4 set_mon_vals no callback set 2024-04-29T20:23:00.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.902+0000 7f2ffecce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.902+0000 7f2ffecce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.902+0000 7f2ffecce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:00.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.902+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd8005080 con 0x7f2ff80b8b90 2024-04-29T20:23:00.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.907+0000 7f0931f63640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.907+0000 7f0931f63640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.907+0000 7f0931f63640 10 monclient: monmap: 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.907919+0000 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.907919+0000 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.907+0000 7f0931f63640 1 Processor -- start 2024-04-29T20:23:00.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 1 -- start start 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient: init 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f092c0b2230 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f092c0aa460 0x7f092c0b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): picked mon.noname-c con 0x7f092c0aa460 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c0b5c60 0x7f092c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): picked mon.noname-b con 0x7f092c0b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f092c0ac1f0 con 0x7f092c0b2230 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f092c0ac4d0 con 0x7f092c0aa460 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f092c0b3440 con 0x7f092c0b5c60 2024-04-29T20:23:00.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.908+0000 7f0931f63640 10 monclient(hunting): authenticate will time out at 1530.068549s 2024-04-29T20:23:00.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.911+0000 7f87c1994640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f87bc0ad6a0 0x7f87bc144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.913+0000 7f092b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f092c0b2230 0x7f092c146970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51878/0 (socket says 172.21.15.64:51878) 2024-04-29T20:23:00.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.913+0000 7f092b7fe640 1 -- 172.21.15.64:0/3837875385 learned_addr learned my addr 172.21.15.64:0/3837875385 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.913+0000 7f3bccb35640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3bc80b5c60 0x7f3bc814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.913+0000 7f87c1994640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f87bc0ad6a0 0x7f87bc144b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34584/0 (socket says 172.21.15.64:34584) 2024-04-29T20:23:00.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f87c1994640 1 -- 172.21.15.64:0/3466282709 learned_addr learned my addr 172.21.15.64:0/3466282709 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f87c1994640 10 monclient(hunting): get_auth_request con 0x7f87bc0ad6a0 auth_method 0 2024-04-29T20:23:00.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f87c1994640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f87c1994640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f87c1994640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f3bccb35640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3bc80b5c60 0x7f3bc814dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34596/0 (socket says 172.21.15.64:34596) 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f3bccb35640 1 -- 172.21.15.64:0/1543252223 learned_addr learned my addr 172.21.15.64:0/1543252223 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f3bccb35640 10 monclient(hunting): get_auth_request con 0x7f3bc80b5c60 auth_method 0 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f3bccb35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f3bccb35640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.914+0000 7f3bccb35640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.915+0000 7f87c1994640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.915+0000 7f87c1994640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.915+0000 7f87c1994640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.916+0000 7f3bccb35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.916+0000 7f3bccb35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.916+0000 7f3bccb35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 1 -- 172.21.15.64:0/3466282709 >> v2:172.21.15.167:3300/0 conn(0x7f87bc14ba90 msgr2=0x7f87bc0abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 1 --2- 172.21.15.64:0/3466282709 >> v2:172.21.15.167:3300/0 conn(0x7f87bc14ba90 0x7f87bc0abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 1 -- 172.21.15.64:0/3466282709 >> v1:172.21.15.167:6789/0 conn(0x7f87bc0aad50 legacy=0x7f87bc14b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.919+0000 7f87c1994640 1 -- 172.21.15.64:0/3466282709 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc0ab2b0 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.921+0000 7f87c1994640 1 --2- 172.21.15.64:0/3466282709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc144b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0b13f0 tx=0x7f87ac02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f139ca0d4d1d19fa server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.923+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1604684460 0 0) 0x7f092c0ac1f0 con 0x7f092c0b2230 2024-04-29T20:23:00.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.923+0000 7f092a7fc640 10 monclient(hunting): _negotiate 2024-04-29T20:23:00.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.923+0000 7f092a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.923+0000 7f092a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.923+0000 7f092a7fc640 10 monclient(hunting): my global_id is 4338 2024-04-29T20:23:00.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.923+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0914003520 con 0x7f092c0b2230 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 1 -- 172.21.15.64:0/1543252223 >> v2:172.21.15.167:3300/0 conn(0x7f3bc80aa460 msgr2=0x7f3bc80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 1 --2- 172.21.15.64:0/1543252223 >> v2:172.21.15.167:3300/0 conn(0x7f3bc80aa460 0x7f3bc80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 1 -- 172.21.15.64:0/1543252223 >> v1:172.21.15.167:6789/0 conn(0x7f3bc80b2230 legacy=0x7f3bc8146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.924+0000 7f3bccb35640 1 -- 172.21.15.64:0/1543252223 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc80aa8d0 con 0x7f3bc80b5c60 2024-04-29T20:23:00.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.929+0000 7f3bccb35640 1 --2- 172.21.15.64:0/1543252223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b5c60 0x7f3bc814dce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3bc806c730 tx=0x7f3bc002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e2d52d10a9fbe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 1 -- 172.21.15.64:0/3466282709 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac038070 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 10 monclient: dump: 2024-04-29T20:23:00.933 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.933 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87c33f7640 5 monclient: authenticate success, global_id 4332 2024-04-29T20:23:00.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.932+0000 7f87c33f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87b3fff640 1 -- 172.21.15.64:0/3466282709 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ac02f050 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87b3fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87b3fff640 1 -- 172.21.15.64:0/3466282709 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac031970 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87b3fff640 10 monclient: dump: 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87c33f7640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.939+0000 7f87c33f7640 4 set_mon_vals no callback set 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 10 monclient: shutdown 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 -- 172.21.15.64:0/3466282709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 msgr2=0x7f87bc144b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 --2- 172.21.15.64:0/3466282709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc144b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0b13f0 tx=0x7f87ac02b7f0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 -- 172.21.15.64:0/3466282709 shutdown_connections 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 --2- 172.21.15.64:0/3466282709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc144b50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 --2- 172.21.15.64:0/3466282709 >> v2:172.21.15.167:3300/0 conn(0x7f87bc14ba90 0x7f87bc0abeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 -- 172.21.15.64:0/3466282709 >> 172.21.15.64:0/3466282709 conn(0x7f87bc0b4b60 msgr2=0x7f87bc0b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2152217249 0 0) 0x7f0914003520 con 0x7f092c0b2230 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c0b5c60 msgr2=0x7f092c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c0b5c60 0x7f092c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 >> v2:172.21.15.167:3300/0 conn(0x7f092c0aa460 msgr2=0x7f092c0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 --2- 172.21.15.64:0/3837875385 >> v2:172.21.15.167:3300/0 conn(0x7f092c0aa460 0x7f092c0b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 monclient: found mon.noname-a-legacy 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.64:6789/0 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092c0aa8d0 con 0x7f092c0b2230 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f0931f63640 5 monclient: authenticate success, global_id 4338 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f0931f63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 -- 172.21.15.64:0/3466282709 shutdown_connections 2024-04-29T20:23:00.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f87c33f7640 1 -- 172.21.15.64:0/3466282709 wait complete. 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 525552804 0 0) 0x7f091c001670 con 0x7f092c0b2230 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: dump: 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.945 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] but i'm connected to v1:172.21.15.64:6789/0, reconnecting 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 >> v1:172.21.15.64:6789/0 conn(0x7f092c0b2230 legacy=0x7f092c146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09140039d0 0x7f0914003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): picked mon.b con 0x7f09140039d0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0914004800 0x7f0914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): picked mon.a con 0x7f0914004800 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c0b5c60 0x7f091400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): picked mon.c con 0x7f092c0b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f092c0ac4d0 con 0x7f0914004800 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f092c0b3440 con 0x7f09140039d0 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f09140055c0 con 0x7f092c0b5c60 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f092a7fc640 10 monclient(hunting): _finish_auth 0 2024-04-29T20:23:00.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.940+0000 7f0931f63640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 10 monclient: monmap: 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 1 librados: starting msgr at 2024-04-29T20:23:00.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 1 librados: starting objecter 2024-04-29T20:23:00.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 1 -- 172.21.15.64:0/1543252223 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc002c9f0 con 0x7f3bc80b5c60 2024-04-29T20:23:00.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 1 Processor -- start 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: dump: 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.950 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 1 -- start start 2024-04-29T20:23:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 1 librados: setting wanted keys 2024-04-29T20:23:00.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 1 librados: calling monclient init 2024-04-29T20:23:00.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f87c33f7640 10 monclient: init 2024-04-29T20:23:00.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 1 -- 172.21.15.64:0/1543252223 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc002cb90 con 0x7f3bc80b5c60 2024-04-29T20:23:00.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 1 -- 172.21.15.64:0/1543252223 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00343f0 con 0x7f3bc80b5c60 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bce598640 5 monclient: authenticate success, global_id 4335 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bce598640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bce598640 4 set_mon_vals no callback set 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.941+0000 7f3bc6ffd640 10 monclient: dump: 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.955 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.955 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:00.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f87bc0aad50 0x7f87bc13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): picked mon.noname-c con 0x7f87bc0aad50 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc13c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): picked mon.noname-a con 0x7f87bc0ad6a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc14ba90 0x7f87bc1404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): picked mon.noname-b con 0x7f87bc14ba90 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc145460 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1452e0 con 0x7f87bc14ba90 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc152130 con 0x7f87bc0aad50 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f87c33f7640 10 monclient(hunting): authenticate will time out at 1530.102018s 2024-04-29T20:23:00.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 10 monclient: shutdown 2024-04-29T20:23:00.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 -- 172.21.15.64:0/1543252223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b5c60 msgr2=0x7f3bc814dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 --2- 172.21.15.64:0/1543252223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b5c60 0x7f3bc814dce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3bc806c730 tx=0x7f3bc002be50 comp rx=0 tx=0).stop 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 -- 172.21.15.64:0/1543252223 shutdown_connections 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 --2- 172.21.15.64:0/1543252223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b5c60 0x7f3bc814dce0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 --2- 172.21.15.64:0/1543252223 >> v2:172.21.15.167:3300/0 conn(0x7f3bc80aa460 0x7f3bc80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 -- 172.21.15.64:0/1543252223 >> 172.21.15.64:0/1543252223 conn(0x7f3bc806c960 msgr2=0x7f3bc806cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 -- 172.21.15.64:0/1543252223 shutdown_connections 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.942+0000 7f3bce598640 1 -- 172.21.15.64:0/1543252223 wait complete. 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.943+0000 7f3bce598640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.943+0000 7f3bce598640 10 monclient: monmap: 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.943+0000 7f3bce598640 1 librados: starting msgr at 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.943+0000 7f3bce598640 1 librados: starting objecter 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 1 Processor -- start 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 1 -- start start 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 1 librados: setting wanted keys 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 1 librados: calling monclient init 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 10 monclient: init 2024-04-29T20:23:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3bc80aa460 0x7f3bc8143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 10 monclient(hunting): picked mon.noname-c con 0x7f3bc80aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.944+0000 7f3bce598640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 0x7f3bc80ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): picked mon.noname-a con 0x7f3bc80b2230 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc80b5c60 0x7f3bc813bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): picked mon.noname-b con 0x7f3bc80b5c60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b6210 con 0x7f3bc80b2230 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b6090 con 0x7f3bc80b5c60 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc814e220 con 0x7f3bc80aa460 2024-04-29T20:23:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f3bce598640 10 monclient(hunting): authenticate will time out at 1530.104768s 2024-04-29T20:23:00.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.945+0000 7f092affd640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0914004800 0x7f0914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.946+0000 7f092affd640 10 monclient(hunting): get_auth_request con 0x7f0914004800 auth_method 0 2024-04-29T20:23:00.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.946+0000 7f092affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.946+0000 7f092affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.946+0000 7f092affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.947+0000 7f092affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.947+0000 7f092affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.947+0000 7f092affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 10 monclient(hunting): handle_auth_done global_id 4338 payload 293 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 1 -- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c0b5c60 msgr2=0x7f091400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c0b5c60 0x7f091400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 1 -- 172.21.15.64:0/3837875385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09140039d0 msgr2=0x7f0914003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09140039d0 0x7f0914003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 1 monclient: found mon.a 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 1 -- 172.21.15.64:0/3837875385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092c0aa8d0 con 0x7f0914004800 2024-04-29T20:23:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7f092affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7ff9c49fe640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7ff9c49fe640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.950+0000 7ff9c49fe640 10 monclient: monmap: 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.951337+0000 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.951337+0000 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.969 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.951+0000 7ff9c49fe640 1 Processor -- start 2024-04-29T20:23:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.951+0000 7ff9c49fe640 1 -- start start 2024-04-29T20:23:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.951+0000 7ff9c49fe640 10 monclient: init 2024-04-29T20:23:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.952+0000 7f87c0992640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc13c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7ff9c00b22a0 0x7ff9c0146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c00b22a0 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9c00b2c20 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9c00aa6a0 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c00ac1f0 con 0x7ff9c00b2c20 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c00ac4d0 con 0x7ff9c00aa6a0 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c00b33d0 con 0x7ff9c00b22a0 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7ff9c49fe640 10 monclient(hunting): authenticate will time out at 1530.112717s 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7f87c0992640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc13c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34606/0 (socket says 172.21.15.64:34606) 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7f87c0992640 1 -- 172.21.15.64:0/2379787752 learned_addr learned my addr 172.21.15.64:0/2379787752 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7f87c0992640 10 monclient(hunting): get_auth_request con 0x7f87bc0ad6a0 auth_method 0 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7f87c0992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7f87c0992640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.953+0000 7f87c0992640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.954+0000 7f87c0992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.954+0000 7f87c0992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.954+0000 7f87c0992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.954+0000 7f092affd640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0914004800 0x7f0914004c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0920009860 tx=0x7f0914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3879fbf8bf33fe35 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 1 -- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f87bc0aad50 msgr2=0x7f87bc13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f87bc0aad50 0x7f87bc13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 1 -- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc14ba90 msgr2=0x7f87bc1404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc14ba90 0x7f87bc1404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f87c0992640 1 -- 172.21.15.64:0/2379787752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc13ca60 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f7a33df4640 10 monclient: get_monmap_and_config 2024-04-29T20:23:00.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f7a33df4640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.958+0000 7f7a33df4640 10 monclient: monmap: 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:00.959585+0000 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:00.959585+0000 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.959+0000 7f7a33df4640 1 Processor -- start 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.959+0000 7f7a33df4640 1 -- start start 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.959+0000 7f7a33df4640 10 monclient: init 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0b2230 0x7f7a2c146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f7a2c0aa460 0x7f7a2c0b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c0aa460 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a2c0b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a2c146eb0 con 0x7f7a2c0b5c60 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2c0ac1f0 con 0x7f7a2c0aa460 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c0ac4d0 con 0x7f7a2c0b2230 2024-04-29T20:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.960+0000 7f7a33df4640 10 monclient(hunting): authenticate will time out at 1530.120111s 2024-04-29T20:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.961+0000 7ff9b7fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff9c00b2c20 0x7ff9c00b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51892/0 (socket says 172.21.15.64:51892) 2024-04-29T20:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.961+0000 7ff9b7fff640 1 -- 172.21.15.64:0/2622329530 learned_addr learned my addr 172.21.15.64:0/2622329530 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.963+0000 7f3bc77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 0x7f3bc80ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f87c0992640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc13c520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b9c0 tx=0x7f87b400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f87b1ffb640 10 client.?.objecter ms_handle_connect 0x7f87bc0ad6a0 2024-04-29T20:23:00.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f87b1ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f3bc77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 0x7f3bc80ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34616/0 (socket says 172.21.15.64:34616) 2024-04-29T20:23:00.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f3bc77fe640 1 -- 172.21.15.64:0/2979995131 learned_addr learned my addr 172.21.15.64:0/2979995131 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:00.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f3bc77fe640 10 monclient(hunting): get_auth_request con 0x7f3bc80b2230 auth_method 0 2024-04-29T20:23:00.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f3bc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f3bc77fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.964+0000 7f3bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.965+0000 7f3bc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.965+0000 7f3bc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.965+0000 7f3bc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 1 -- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3bc80aa460 msgr2=0x7f3bc8143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3bc80aa460 0x7f3bc8143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 1 -- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc80b5c60 msgr2=0x7f3bc813bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc80b5c60 0x7f3bc813bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 1 monclient: found mon.noname-a 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.970+0000 7f3bc77fe640 1 -- 172.21.15.64:0/2979995131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc80ae910 con 0x7f3bc80b2230 2024-04-29T20:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.972+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f363c047070 con 0x7f36440b2ee0 2024-04-29T20:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.972+0000 7f363a7fc640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f362806e2d0 0x7f3628070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.973+0000 7f36496a8640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f362806e2d0 0x7f3628070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.973+0000 7f36496a8640 10 monclient: get_auth_request con 0x7f362806e2d0 auth_method 0 2024-04-29T20:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.973+0000 7f363a7fc640 10 client.4259.objecter ms_handle_connect 0x7f362806e2d0 2024-04-29T20:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.973+0000 7f36496a8640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f362806e2d0 0x7f3628070790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f36441501d0 tx=0x7f3640007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4019070 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 10 monclient: dump: 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87c33f7640 5 monclient: authenticate success, global_id 4341 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87c33f7640 10 monclient: _renew_subs 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87c33f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.974+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc13e8a0 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.975+0000 7f87c33f7640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.975+0000 7f87c33f7640 10 monclient: _renew_subs 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.975+0000 7f87c33f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.975+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc140a00 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.975+0000 7f87c33f7640 1 librados: init done 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.976+0000 7f87c33f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.976+0000 7f87c33f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.976+0000 7f87c33f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:00.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.976+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc04fce0 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985220798 0 0) 0x7ff9c00ac1f0 con 0x7ff9c00b2c20 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7ff9bdd9b640 10 monclient(hunting): _negotiate 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7ff9bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7ff9bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7ff9bdd9b640 10 monclient(hunting): my global_id is 4347 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ac003520 con 0x7ff9c00b2c20 2024-04-29T20:23:00.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7f3bc77fe640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 0x7f3bc80ae3d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b440 tx=0x7f3bbc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7f3bc4ff9640 10 client.?.objecter ms_handle_connect 0x7f3bc80b2230 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.978+0000 7f3bc4ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b400cba0 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b1ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4023060 con 0x7f87bc0ad6a0 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b1ffb640 10 monclient: dump: 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.985 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f87b37fe640 4 set_mon_vals no callback set 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f7a32391640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f7a2c0b5c60 0x7f7a2c0b37b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51898/0 (socket says 172.21.15.64:51898) 2024-04-29T20:23:00.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.979+0000 7f7a32391640 1 -- 172.21.15.64:0/1761661618 learned_addr learned my addr 172.21.15.64:0/1761661618 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920037070 con 0x7f0914004800 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: dump: 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.988 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0920032620 con 0x7f0914004800 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f0931f63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 1 -- 172.21.15.64:0/3837875385 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920009460 con 0x7f0914004800 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f092a7fc640 10 monclient: dump: 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.989 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.990 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.990 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.990 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.990 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f0931f63640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:00.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.986+0000 7f0931f63640 4 set_mon_vals no callback set 2024-04-29T20:23:00.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 10 monclient: shutdown 2024-04-29T20:23:00.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:00.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 1 -- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0914004800 msgr2=0x7f0914004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 1 --2- 172.21.15.64:0/3837875385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0914004800 0x7f0914004c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0920009860 tx=0x7f0914005830 comp rx=0 tx=0).stop 2024-04-29T20:23:00.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f092bfff640 1 -- 172.21.15.64:0/3837875385 reap_dead start 2024-04-29T20:23:00.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 1 -- 172.21.15.64:0/3837875385 shutdown_connections 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 1 -- 172.21.15.64:0/3837875385 >> 172.21.15.64:0/3837875385 conn(0x7f092c06c960 msgr2=0x7f092c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 1 -- 172.21.15.64:0/3837875385 shutdown_connections 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.987+0000 7f0931f63640 1 -- 172.21.15.64:0/3837875385 wait complete. 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 10 monclient: build_initial_monmap 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 10 monclient: monmap: 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:00.992 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 1 librados: starting msgr at 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 1 librados: starting objecter 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 1 Processor -- start 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 1 -- start start 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 1 librados: setting wanted keys 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 1 librados: calling monclient init 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.988+0000 7f0931f63640 10 monclient: init 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c1484b0 0x7f092c13abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): picked mon.noname-c con 0x7f092c1484b0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 0x7f092c1492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): picked mon.noname-a con 0x7f092c148890 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f092c14acc0 0x7f092c139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): picked mon.noname-b con 0x7f092c14acc0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f092c0b2790 con 0x7f092c148890 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f092c0b2610 con 0x7f092c14acc0 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f092c14bcd0 con 0x7f092c1484b0 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.989+0000 7f0931f63640 10 monclient(hunting): authenticate will time out at 1530.149105s 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 812405725 0 0) 0x7ff9ac003520 con 0x7ff9c00b2c20 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 >> v2:172.21.15.167:3300/0 conn(0x7ff9c00b22a0 msgr2=0x7ff9c0146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 --2- 172.21.15.64:0/2622329530 >> v2:172.21.15.167:3300/0 conn(0x7ff9c00b22a0 0x7ff9c0146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 >> v1:172.21.15.167:6789/0 conn(0x7ff9c00aa6a0 legacy=0x7ff9c014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: _finish_hunting 0 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 monclient: found mon.noname-a-legacy 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.64:6789/0 2024-04-29T20:23:00.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c00b3040 con 0x7ff9c00b2c20 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: _finish_auth 0 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: _check_auth_tickets 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 525552804 0 0) 0x7ff9a8001670 con 0x7ff9c00b2c20 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: dump: 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:00.996 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] but i'm connected to v1:172.21.15.64:6789/0, reconnecting 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 >> v1:172.21.15.64:6789/0 conn(0x7ff9c00b2c20 legacy=0x7ff9c00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0039b0 0x7ff9ac003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): picked mon.a con 0x7ff9ac0039b0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9ac004800 0x7ff9ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): picked mon.c con 0x7ff9ac004800 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9ac0052e0 0x7ff9ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): picked mon.b con 0x7ff9ac0052e0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c00b33d0 con 0x7ff9ac0039b0 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac00c1e0 con 0x7ff9ac0052e0 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ac00c3e0 con 0x7ff9ac004800 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.991+0000 7ff9bdd9b640 10 monclient(hunting): _finish_auth 0 2024-04-29T20:23:00.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.996+0000 7f092affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 0x7f092c1492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc015020 con 0x7f3bc80b2230 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: dump: 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.001 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: _finish_auth 0 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bbc0118b0 con 0x7f3bc80b2230 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc011ca0 con 0x7f3bc80b2230 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:01.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc4ff9640 10 monclient: dump: 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 5 monclient: authenticate success, global_id 4344 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 10 monclient: _renew_subs 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc80adac0 con 0x7f3bc80b2230 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bc67fc640 4 set_mon_vals no callback set 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 10 monclient: _renew_subs 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc813c640 con 0x7f3bc80b2230 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:00.999+0000 7f3bce598640 1 librados: init done 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f092affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 0x7f092c1492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34618/0 (socket says 172.21.15.64:34618) 2024-04-29T20:23:01.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f092affd640 1 -- 172.21.15.64:0/1203656479 learned_addr learned my addr 172.21.15.64:0/1203656479 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f092affd640 10 monclient(hunting): get_auth_request con 0x7f092c148890 auth_method 0 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f092affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f092affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f092affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.000+0000 7f3bce598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7f3bce598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7f3bce598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc804fce0 con 0x7f3bc80b2230 2024-04-29T20:23:01.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7f092affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7f092affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7f092affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:01.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.001+0000 7ff9be59c640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0039b0 0x7ff9ac003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 1 -- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c1484b0 msgr2=0x7f092c13abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c1484b0 0x7f092c13abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 1 -- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f092c14acc0 msgr2=0x7f092c139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f092c14acc0 0x7f092c139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 10 monclient: _finish_hunting 0 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 1 monclient: found mon.noname-a 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.004+0000 7f092affd640 1 -- 172.21.15.64:0/1203656479 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092c14a630 con 0x7f092c148890 2024-04-29T20:23:01.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.007+0000 7ff9be59c640 10 monclient(hunting): get_auth_request con 0x7ff9ac0039b0 auth_method 0 2024-04-29T20:23:01.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.007+0000 7ff9be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.007+0000 7ff9be59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.007+0000 7ff9be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:23:01.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.158+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f46e0005d10 con 0x7f46e8150200 2024-04-29T20:23:01.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.158+0000 7f46e57fa640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f46cc06e2d0 0x7f46cc070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.158+0000 7f46ecb36640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f46cc06e2d0 0x7f46cc070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.158+0000 7f46ecb36640 10 monclient: get_auth_request con 0x7f46cc06e2d0 auth_method 0 2024-04-29T20:23:01.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.159+0000 7f46e57fa640 10 client.4348.objecter ms_handle_connect 0x7f46cc06e2d0 2024-04-29T20:23:01.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.159+0000 7f46ecb36640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f46cc06e2d0 0x7f46cc070790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46d4004f60 tx=0x7f46d4007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.163+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f46e0057f00 con 0x7f46e8150200 2024-04-29T20:23:01.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.163+0000 7f46e57fa640 10 client.4348.objecter ms_dispatch 0x7f46e8000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.163+0000 7f46e57fa640 3 client.4348.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.163+0000 7f46e57fa640 3 client.4348.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.163+0000 7f46e57fa640 20 client.4348.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.177+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46e0024020 con 0x7f46e8150200 2024-04-29T20:23:01.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.177+0000 7f46e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.177+0000 7f46e57fa640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.184+0000 7f7a3138f640 1 --2- 172.21.15.64:0/1761661618 >> v2:172.21.15.167:3300/0 conn(0x7f7a2c0aa460 0x7f7a2c0b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.185+0000 7ff9bed9d640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9ac0052e0 0x7ff9ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7f7a3138f640 10 monclient(hunting): get_auth_request con 0x7f7a2c0aa460 auth_method 0 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7f7a3138f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7f7a3138f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7f7a3138f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7ff9bed9d640 10 monclient(hunting): get_auth_request con 0x7ff9ac0052e0 auth_method 0 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7ff9bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7ff9bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.186+0000 7ff9bed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:23:01.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.187+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.187+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.187+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-29T20:23:01.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.187+0000 7f7a3138f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.187+0000 7f7a3138f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.187+0000 7f7a3138f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:01.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-04-29T20:23:01.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 1 -- 172.21.15.64:0/1761661618 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0b2230 msgr2=0x7f7a2c146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 1 --2- 172.21.15.64:0/1761661618 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0b2230 0x7f7a2c146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 1 -- 172.21.15.64:0/1761661618 >> v1:172.21.15.64:6789/0 conn(0x7f7a2c0b5c60 legacy=0x7f7a2c0b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 10 monclient: _finish_hunting 0 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 1 monclient: found mon.noname-c 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.190+0000 7f7a3138f640 1 -- 172.21.15.64:0/1761661618 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c0b3440 con 0x7f7a2c0aa460 2024-04-29T20:23:01.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.192+0000 7f7a3138f640 1 --2- 172.21.15.64:0/1761661618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0aa460 0x7f7a2c0b2f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0097a0 tx=0x7f7a1c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f33560689dd3727 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_done global_id 4347 payload 293 2024-04-29T20:23:01.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 1 -- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9ac004800 msgr2=0x7ff9ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:01.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9ac004800 0x7ff9ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 1 -- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0039b0 msgr2=0x7ff9ac003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0039b0 0x7ff9ac003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 10 monclient: _finish_hunting 0 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 1 monclient: found mon.b 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 1 -- 172.21.15.64:0/2622329530 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c00b3040 con 0x7ff9ac0052e0 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 10 monclient: _finish_auth 0 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 10 monclient: _check_auth_tickets 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9bed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9c49fe640 5 monclient: authenticate success, global_id 4347 2024-04-29T20:23:01.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.194+0000 7ff9c49fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:01.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.195+0000 7ff9bed9d640 1 -- 172.21.15.64:0/2622329530 reap_dead start 2024-04-29T20:23:01.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.197+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f363c07a510 con 0x7f36440b2ee0 2024-04-29T20:23:01.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.197+0000 7f363a7fc640 10 client.4259.objecter ms_dispatch 0x7f3644000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.197+0000 7f363a7fc640 3 client.4259.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.197+0000 7f363a7fc640 3 client.4259.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.197+0000 7f363a7fc640 20 client.4259.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.216+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f363c04e070 con 0x7f36440b2ee0 2024-04-29T20:23:01.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.216+0000 7f363a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.216+0000 7f363a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.309+0000 7f092affd640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 0x7f092c1492f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0920009980 tx=0x7f09200047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.309+0000 7f0913fff640 10 client.?.objecter ms_handle_connect 0x7f092c148890 2024-04-29T20:23:01.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.309+0000 7f0913fff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:01.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.325+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87b401c070 con 0x7f87bc0ad6a0 2024-04-29T20:23:01.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.326+0000 7f87b1ffb640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f87a006e280 0x7f87a0070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.326+0000 7f87c1193640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f87a006e280 0x7f87a0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.326+0000 7f87c1193640 10 monclient: get_auth_request con 0x7f87a006e280 auth_method 0 2024-04-29T20:23:01.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.326+0000 7f87b1ffb640 10 client.4341.objecter ms_handle_connect 0x7f87a006e280 2024-04-29T20:23:01.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.326+0000 7f87c1193640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f87a006e280 0x7f87a0070740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f87bc069370 tx=0x7f87b8006ba0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f87b4057f10 con 0x7f87bc0ad6a0 2024-04-29T20:23:01.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 10 client.4341.objecter ms_dispatch 0x7f87bc000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 3 client.4341.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 3 client.4341.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 20 client.4341.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87b4059e70 con 0x7f87bc0ad6a0 2024-04-29T20:23:01.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.342+0000 7f87b1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920037070 con 0x7f092c148890 2024-04-29T20:23:01.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:01.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: dump: 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.368 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: _finish_auth 0 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0920038040 con 0x7f092c148890 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09200035f0 con 0x7f092c148890 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0913fff640 10 monclient: dump: 2024-04-29T20:23:01.369 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 5 monclient: authenticate success, global_id 4350 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 10 monclient: _renew_subs 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0929ffb640 4 set_mon_vals no callback set 2024-04-29T20:23:01.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092c148e10 con 0x7f092c148890 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 10 monclient: _renew_subs 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f092c14b320 con 0x7f092c148890 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.362+0000 7f0931f63640 1 librados: init done 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.363+0000 7f0931f63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.363+0000 7f0931f63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.363+0000 7f0931f63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.363+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f092c04fce0 con 0x7f092c148890 2024-04-29T20:23:01.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.390+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3bbc018020 con 0x7f3bc80b2230 2024-04-29T20:23:01.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.390+0000 7f3bc4ff9640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3bb406e2d0 0x7f3bb4070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.391+0000 7f3bc7fff640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3bb406e2d0 0x7f3bb4070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.391+0000 7f3bc7fff640 10 monclient: get_auth_request con 0x7f3bb406e2d0 auth_method 0 2024-04-29T20:23:01.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.391+0000 7f3bc4ff9640 10 client.4344.objecter ms_handle_connect 0x7f3bb406e2d0 2024-04-29T20:23:01.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.391+0000 7f3bc7fff640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3bb406e2d0 0x7f3bb4070790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8001560 tx=0x7f3bb8007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.409+0000 7f46eed9a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-29T20:23:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.409+0000 7f46eed9a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-29T20:23:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.409+0000 7f46eed9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.409+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f46c8005640 con 0x7f46e8150200 2024-04-29T20:23:01.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.459+0000 7f364c10d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-29T20:23:01.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.459+0000 7f364c10d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-29T20:23:01.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.459+0000 7f364c10d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:01.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.459+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f364404fce0 con 0x7f36440b2ee0 2024-04-29T20:23:01.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.464+0000 7f363a7fc640 1 -- 172.21.15.64:0/2981763944 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f363c07cb70 con 0x7f36440b2ee0 2024-04-29T20:23:01.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.464+0000 7f363a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-29T20:23:01.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.464+0000 7f363a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:01.466 INFO:teuthology.orchestra.run.smithi064.stdout:64424509445 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 10 librados: watch_flush enter 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 10 librados: watch_flush exit 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 20 client.4259.objecter shutdown clearing up homeless session... 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 10 client.4259.objecter successfully canceled tick 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f362806e2d0 msgr2=0x7f3628070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f362806e2d0 0x7f3628070790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f36441501d0 tx=0x7f3640007d30 comp rx=0 tx=0).stop 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 10 monclient: shutdown 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 msgr2=0x7f3644150850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 0x7f3644150850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c004a40 comp rx=0 tx=0).stop 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 shutdown_connections 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f362806e2d0 0x7f3628070790 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3644150200 0x7f364412f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36440b37b0 0x7f36441351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 --2- 172.21.15.64:0/2981763944 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36440b2ee0 0x7f3644150850 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 >> 172.21.15.64:0/2981763944 conn(0x7f364406c960 msgr2=0x7f3644070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 shutdown_connections 2024-04-29T20:23:01.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 -- 172.21.15.64:0/2981763944 wait complete. 2024-04-29T20:23:01.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.467+0000 7f364c10d640 1 librados: shutdown 2024-04-29T20:23:01.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.473+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fe801a070 con 0x7f2ff80b8b90 2024-04-29T20:23:01.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.473+0000 7f2ff57fa640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2fe406e300 0x7f2fe40707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.473+0000 7f2ffca6a640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2fe406e300 0x7f2fe40707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.473+0000 7f2ffca6a640 10 monclient: get_auth_request con 0x7f2fe406e300 auth_method 0 2024-04-29T20:23:01.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.474+0000 7f2ff57fa640 10 client.4351.objecter ms_handle_connect 0x7f2fe406e300 2024-04-29T20:23:01.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.474+0000 7f2ffca6a640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2fe406e300 0x7f2fe40707c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00017d0 tx=0x7f2ff0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.477+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f2fe80570f0 con 0x7f2ff80b8b90 2024-04-29T20:23:01.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.477+0000 7f2ff57fa640 10 client.4351.objecter ms_dispatch 0x7f2ff8000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.477+0000 7f2ff57fa640 3 client.4351.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.477+0000 7f2ff57fa640 3 client.4351.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.477+0000 7f2ff57fa640 20 client.4351.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.2 2024-04-29T20:23:01.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.488+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2fe8021020 con 0x7f2ff80b8b90 2024-04-29T20:23:01.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.488+0000 7f2ff57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.488+0000 7f2ff57fa640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.488+0000 7ff9bed9d640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9ac0052e0 0x7ff9ac00b9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8001d20 tx=0x7ff9ac005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1505c0fa856a4f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 1 -- 172.21.15.64:0/1761661618 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0315a0 con 0x7f7a2c0aa460 2024-04-29T20:23:01.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: dump: 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: _finish_auth 0 2024-04-29T20:23:01.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: _check_auth_tickets 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 1 -- 172.21.15.64:0/1761661618 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1c037070 con 0x7f7a2c0aa460 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a33df4640 5 monclient: authenticate success, global_id 4354 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a33df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a33df4640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 1 -- 172.21.15.64:0/1761661618 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c031960 con 0x7f7a2c0aa460 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a33df4640 4 set_mon_vals no callback set 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.503+0000 7f7a30b8e640 10 monclient: dump: 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.506 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.507 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 10 monclient: shutdown 2024-04-29T20:23:01.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 -- 172.21.15.64:0/1761661618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0aa460 msgr2=0x7f7a2c0b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 --2- 172.21.15.64:0/1761661618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0aa460 0x7f7a2c0b2f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0097a0 tx=0x7f7a1c02b840 comp rx=0 tx=0).stop 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 -- 172.21.15.64:0/1761661618 shutdown_connections 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 --2- 172.21.15.64:0/1761661618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0aa460 0x7f7a2c0b2f00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 --2- 172.21.15.64:0/1761661618 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0b2230 0x7f7a2c146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 -- 172.21.15.64:0/1761661618 >> 172.21.15.64:0/1761661618 conn(0x7f7a2c06c960 msgr2=0x7f7a2c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 -- 172.21.15.64:0/1761661618 shutdown_connections 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.504+0000 7f7a33df4640 1 -- 172.21.15.64:0/1761661618 wait complete. 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.505+0000 7f7a33df4640 10 monclient: build_initial_monmap 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.505+0000 7f7a33df4640 10 monclient: monmap: 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:01.509 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.505+0000 7f7a33df4640 1 librados: starting msgr at 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.505+0000 7f7a33df4640 1 librados: starting objecter 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7f7a33df4640 1 Processor -- start 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7f7a33df4640 1 -- start start 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7f7a33df4640 1 librados: setting wanted keys 2024-04-29T20:23:01.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7f7a33df4640 1 librados: calling monclient init 2024-04-29T20:23:01.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7f7a33df4640 10 monclient: init 2024-04-29T20:23:01.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8001920 con 0x7ff9ac0052e0 2024-04-29T20:23:01.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:01.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: dump: 2024-04-29T20:23:01.511 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b8003970 con 0x7ff9ac0052e0 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9c49fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9c49fe640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 1 -- 172.21.15.64:0/2622329530 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8003d40 con 0x7ff9ac0052e0 2024-04-29T20:23:01.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9c49fe640 4 set_mon_vals no callback set 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7ff9bdd9b640 10 monclient: dump: 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.513 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.506+0000 7f7a33df4640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:01.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:01.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 0x7f7a2c143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c0aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:01.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a2c0b2230 0x7f7a2c1dbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 10 monclient: shutdown 2024-04-29T20:23:01.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:01.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): picked mon.noname-a con 0x7f7a2c0b2230 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:01.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 -- 172.21.15.64:0/2622329530 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9ac0052e0 msgr2=0x7ff9ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9ac0052e0 0x7ff9ac00b9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8001d20 tx=0x7ff9ac005830 comp rx=0 tx=0).stop 2024-04-29T20:23:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b5c60 0x7f7a2c13b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0b5c60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c0b6250 con 0x7f7a2c0b2230 2024-04-29T20:23:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c0b60d0 con 0x7f7a2c0b5c60 2024-04-29T20:23:01.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:01.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 -- 172.21.15.64:0/2622329530 shutdown_connections 2024-04-29T20:23:01.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/2622329530 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9ac0052e0 0x7ff9ac00b9c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 -- 172.21.15.64:0/2622329530 >> 172.21.15.64:0/2622329530 conn(0x7ff9c006c940 msgr2=0x7ff9c006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:01.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c0b2610 con 0x7f7a2c0aa460 2024-04-29T20:23:01.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:01.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a33df4640 10 monclient(hunting): authenticate will time out at 1530.666985s 2024-04-29T20:23:01.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 -- 172.21.15.64:0/2622329530 shutdown_connections 2024-04-29T20:23:01.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7ff9c49fe640 1 -- 172.21.15.64:0/2622329530 wait complete. 2024-04-29T20:23:01.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.507+0000 7f7a31b90640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 0x7f7a2c143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f46e57fa640 1 -- 172.21.15.64:0/1487215457 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f46e0059e60 con 0x7f46e8150200 2024-04-29T20:23:01.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f46e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-29T20:23:01.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f46e57fa640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:01.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f7a31b90640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 0x7f7a2c143a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36800/0 (socket says 172.21.15.64:36800) 2024-04-29T20:23:01.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f7a31b90640 1 -- 172.21.15.64:0/2530329078 learned_addr learned my addr 172.21.15.64:0/2530329078 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f7a31b90640 10 monclient(hunting): get_auth_request con 0x7f7a2c0aa460 auth_method 0 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f7a31b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f7a31b90640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7f7a31b90640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7ff9c49fe640 10 monclient: build_initial_monmap 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.508+0000 7ff9c49fe640 10 monclient: monmap: 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:01.520 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.521 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:01.521 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:01.521 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:01.521 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.521 INFO:teuthology.orchestra.run.smithi064.stdout:64424509447 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 1 librados: starting msgr at 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 1 librados: starting objecter 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 1 Processor -- start 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 1 -- start start 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a31b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a31b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a31b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a32391640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b5c60 0x7f7a2c13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 1 librados: setting wanted keys 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 1 librados: calling monclient init 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7ff9c49fe640 10 monclient: init 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a32391640 10 monclient(hunting): get_auth_request con 0x7f7a2c0b5c60 auth_method 0 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a32391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a32391640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.509+0000 7f7a32391640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:01.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:01.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:23:01.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0052e0 0x7ff9c0145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff9ac0052e0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:01.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 0x7ff9c01d8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7f46eed9a640 10 librados: watch_flush enter 2024-04-29T20:23:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7f46eed9a640 10 librados: watch_flush exit 2024-04-29T20:23:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c00b8790 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9c01d8970 0x7ff9c01db850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7f46eed9a640 20 client.4348.objecter shutdown clearing up homeless session... 2024-04-29T20:23:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7f46eed9a640 10 client.4348.objecter successfully canceled tick 2024-04-29T20:23:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c01d8970 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c00aab30 con 0x7ff9ac0052e0 2024-04-29T20:23:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c00b2890 con 0x7ff9c00b8790 2024-04-29T20:23:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c00aacb0 con 0x7ff9c01d8970 2024-04-29T20:23:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7ff9c49fe640 10 monclient(hunting): authenticate will time out at 1530.670516s 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.510+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f46cc06e2d0 msgr2=0x7f46cc070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f46cc06e2d0 0x7f46cc070790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46d4004f60 tx=0x7f46d4007e30 comp rx=0 tx=0).stop 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 10 monclient: shutdown 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 msgr2=0x7f46e81e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 0x7f46e81e5df0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46e000eed0 tx=0x7f46e000b480 comp rx=0 tx=0).stop 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f7a32391640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f7a32391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f7a32391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:01.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 shutdown_connections 2024-04-29T20:23:01.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f46cc06e2d0 0x7f46cc070790 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e8150200 0x7f46e81e5df0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46e80b37b0 0x7f46e8148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 --2- 172.21.15.64:0/1487215457 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f46e80b2ee0 0x7f46e80b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 >> 172.21.15.64:0/1487215457 conn(0x7f46e806c960 msgr2=0x7f46e813f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 shutdown_connections 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 -- 172.21.15.64:0/1487215457 wait complete. 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.511+0000 7f46eed9a640 1 librados: shutdown 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-04-29T20:23:01.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 1 -- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b5c60 msgr2=0x7f7a2c13b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b5c60 0x7f7a2c13b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 1 -- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a2c0b2230 msgr2=0x7f7a2c1dbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a2c0b2230 0x7f7a2c1dbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 10 monclient: _finish_hunting 0 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 1 monclient: found mon.noname-c 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.514+0000 7f7a31b90640 1 -- 172.21.15.64:0/2530329078 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c0ad9c0 con 0x7f7a2c0aa460 2024-04-29T20:23:01.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.518+0000 7ff9bed9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9c01d8970 0x7ff9c01db850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.518+0000 7ff9b7fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 0x7ff9c01d8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7f7a31b90640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 0x7f7a2c143a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800b4a0 tx=0x7f7a2800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 0x7ff9c01d8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33712/0 (socket says 172.21.15.64:33712) 2024-04-29T20:23:01.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 1 -- 172.21.15.64:0/274798020 learned_addr learned my addr 172.21.15.64:0/274798020 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:01.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): get_auth_request con 0x7ff9c00b8790 auth_method 0 2024-04-29T20:23:01.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:01.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7f7a227fc640 10 client.?.objecter ms_handle_connect 0x7f7a2c0aa460 2024-04-29T20:23:01.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7f7a227fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:01.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9bed9d640 10 monclient(hunting): get_auth_request con 0x7ff9c01d8970 auth_method 0 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.519+0000 7ff9bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.520+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.520+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.520+0000 7ff9bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 1 -- 172.21.15.64:0/274798020 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9c01d8970 msgr2=0x7ff9c01db850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9c01d8970 0x7ff9c01db850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 1 -- 172.21.15.64:0/274798020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0052e0 msgr2=0x7ff9c0145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0052e0 0x7ff9c0145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 10 monclient: _finish_hunting 0 2024-04-29T20:23:01.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 1 monclient: found mon.noname-b 2024-04-29T20:23:01.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:01.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.522+0000 7ff9b7fff640 1 -- 172.21.15.64:0/274798020 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c00b8f80 con 0x7ff9c00b8790 2024-04-29T20:23:01.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.5 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.524+0000 7ff9b7fff640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 0x7ff9c01d8430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c800 tx=0x7ff9a800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.524+0000 7ff9b6ffd640 10 client.?.objecter ms_handle_connect 0x7ff9c00b8790 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.524+0000 7ff9b6ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8018070 con 0x7ff9c00b8790 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: dump: 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.538 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a80074e0 con 0x7ff9c00b8790 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8004a20 con 0x7ff9c00b8790 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9b6ffd640 10 monclient: dump: 2024-04-29T20:23:01.539 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 5 monclient: authenticate success, global_id 4360 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 10 monclient: _renew_subs 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c00b8bb0 con 0x7ff9c00b8790 2024-04-29T20:23:01.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9bd59a640 4 set_mon_vals no callback set 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 10 monclient: _renew_subs 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c00b7e40 con 0x7ff9c00b8790 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.528+0000 7ff9c49fe640 1 librados: init done 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.529+0000 7ff9c49fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.529+0000 7ff9c49fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.529+0000 7ff9c49fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.529+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c004fce0 con 0x7ff9c00b8790 2024-04-29T20:23:01.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28014070 con 0x7f7a2c0aa460 2024-04-29T20:23:01.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: dump: 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: _finish_auth 0 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: _check_auth_tickets 2024-04-29T20:23:01.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a28010610 con 0x7f7a2c0aa460 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28010a00 con 0x7f7a2c0aa460 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a227fc640 10 monclient: dump: 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:01.544 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 5 monclient: authenticate success, global_id 4265 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 10 monclient: _renew_subs 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c0b0fd0 con 0x7f7a2c0aa460 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a23fff640 4 set_mon_vals no callback set 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 10 monclient: _renew_subs 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c0b1b40 con 0x7f7a2c0aa460 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.534+0000 7f7a33df4640 1 librados: init done 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.535+0000 7f7a33df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.535+0000 7f7a33df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.535+0000 7f7a33df4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:01.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.535+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c04fce0 con 0x7f7a2c0aa460 2024-04-29T20:23:01.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.558+0000 7f87c33f7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-29T20:23:01.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.558+0000 7f87c33f7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-29T20:23:01.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.558+0000 7f87c33f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.558+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f87bc0ac9e0 con 0x7f87bc0ad6a0 2024-04-29T20:23:01.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.564+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a28017070 con 0x7f7a2c0aa460 2024-04-29T20:23:01.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.565+0000 7f7a227fc640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f7a1006e280 0x7f7a10070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.565+0000 7f7a3138f640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f7a1006e280 0x7f7a10070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.565+0000 7f7a3138f640 10 monclient: get_auth_request con 0x7f7a1006e280 auth_method 0 2024-04-29T20:23:01.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.565+0000 7f7a227fc640 10 client.4265.objecter ms_handle_connect 0x7f7a1006e280 2024-04-29T20:23:01.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.565+0000 7f7a3138f640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f7a1006e280 0x7f7a10070740 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c036000 tx=0x7f7a1c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.692+0000 7f2ffecce640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-29T20:23:01.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.692+0000 7f2ffecce640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-29T20:23:01.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.692+0000 7f2ffecce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:01.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.692+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2fd8005640 con 0x7f2ff80b8b90 2024-04-29T20:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.710+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f3bbc057fb0 con 0x7f3bc80b2230 2024-04-29T20:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.710+0000 7f3bc4ff9640 10 client.4344.objecter ms_dispatch 0x7f3bc8000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.710+0000 7f3bc4ff9640 3 client.4344.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.710+0000 7f3bc4ff9640 3 client.4344.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.710+0000 7f3bc4ff9640 20 client.4344.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.711+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3bbc01f020 con 0x7f3bc80b2230 2024-04-29T20:23:01.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.711+0000 7f3bc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.711+0000 7f3bc4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.794+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9a8004bc0 con 0x7ff9c00b8790 2024-04-29T20:23:01.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.794+0000 7ff9b6ffd640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff9a006e2b0 0x7ff9a0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:01.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.795+0000 7ff9be59c640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff9a006e2b0 0x7ff9a0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:01.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.795+0000 7ff9be59c640 10 monclient: get_auth_request con 0x7ff9a006e2b0 auth_method 0 2024-04-29T20:23:01.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.795+0000 7ff9b6ffd640 10 client.4360.objecter ms_handle_connect 0x7ff9a006e2b0 2024-04-29T20:23:01.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.795+0000 7ff9be59c640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff9a006e2b0 0x7ff9a0070770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0006b40 tx=0x7ff9b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:01.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.798+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7ff9a8057250 con 0x7ff9c00b8790 2024-04-29T20:23:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.798+0000 7ff9b6ffd640 10 client.4360.objecter ms_dispatch 0x7ff9c0000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.798+0000 7ff9b6ffd640 3 client.4360.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.798+0000 7ff9b6ffd640 3 client.4360.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.798+0000 7ff9b6ffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.808+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9a80591b0 con 0x7ff9c00b8790 2024-04-29T20:23:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.808+0000 7ff9b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.808+0000 7ff9b6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.809+0000 7f2ff57fa640 1 -- 172.21.15.64:0/1169751715 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2fe8059750 con 0x7f2ff80b8b90 2024-04-29T20:23:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.809+0000 7f2ff57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-29T20:23:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.809+0000 7f2ff57fa640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:01.811 INFO:teuthology.orchestra.run.smithi064.stdout:64424509447 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.811+0000 7f2ffecce640 10 librados: watch_flush enter 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 10 librados: watch_flush exit 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 20 client.4351.objecter shutdown clearing up homeless session... 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 10 client.4351.objecter successfully canceled tick 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2fe406e300 msgr2=0x7f2fe40707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2fe406e300 0x7f2fe40707c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00017d0 tx=0x7f2ff0009040 comp rx=0 tx=0).stop 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 10 monclient: shutdown 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 msgr2=0x7f2ff80b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 0x7f2ff80b0d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800c430 tx=0x7f2fe800c8f0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 shutdown_connections 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2fe406e300 0x7f2fe40707c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2ff80b12a0 0x7f2ff81400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ff80b8b90 0x7f2ff80b0d60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 --2- 172.21.15.64:0/1169751715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ff80aa940 0x7f2ff80acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 >> 172.21.15.64:0/1169751715 conn(0x7f2ff8067720 msgr2=0x7f2ff81461c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 shutdown_connections 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 -- 172.21.15.64:0/1169751715 wait complete. 2024-04-29T20:23:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.812+0000 7f2ffecce640 1 librados: shutdown 2024-04-29T20:23:01.828 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.4 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f7a28057830 con 0x7f7a2c0aa460 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 10 client.4265.objecter ms_dispatch 0x7f7a2c000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 3 client.4265.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 3 client.4265.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 20 client.4265.objecter dump_active .. 0 homeless 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a28059790 con 0x7f7a2c0aa460 2024-04-29T20:23:01.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:01.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.849+0000 7f7a227fc640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:01.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.902+0000 7f3bce598640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-29T20:23:01.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.902+0000 7f3bce598640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-29T20:23:01.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.902+0000 7f3bce598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:01.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:01.902+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3bc80adcf0 con 0x7f3bc80b2230 2024-04-29T20:23:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.016+0000 7ff9c49fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-29T20:23:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.016+0000 7ff9c49fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-29T20:23:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.016+0000 7ff9c49fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.016+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff9c00b9150 con 0x7ff9c00b8790 2024-04-29T20:23:02.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.020+0000 7ff9b6ffd640 1 -- 172.21.15.64:0/274798020 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff9a805d4b0 con 0x7ff9c00b8790 2024-04-29T20:23:02.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.020+0000 7ff9b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-29T20:23:02.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.020+0000 7ff9b6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:02.022 INFO:teuthology.orchestra.run.smithi064.stdout:64424509445 2024-04-29T20:23:02.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 10 librados: watch_flush enter 2024-04-29T20:23:02.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 10 librados: watch_flush exit 2024-04-29T20:23:02.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 20 client.4360.objecter shutdown clearing up homeless session... 2024-04-29T20:23:02.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 10 client.4360.objecter successfully canceled tick 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff9a006e2b0 msgr2=0x7ff9a0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff9a006e2b0 0x7ff9a0070770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0006b40 tx=0x7ff9b0008040 comp rx=0 tx=0).stop 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 10 monclient: shutdown 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 msgr2=0x7ff9c01d8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 0x7ff9c01d8430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c800 tx=0x7ff9a800ccc0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 shutdown_connections 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff9a006e2b0 0x7ff9a0070770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff9c01d8970 0x7ff9c01db850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9c00b8790 0x7ff9c01d8430 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 --2- 172.21.15.64:0/274798020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9ac0052e0 0x7ff9c0145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 >> 172.21.15.64:0/274798020 conn(0x7ff9c006c940 msgr2=0x7ff9c00b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 shutdown_connections 2024-04-29T20:23:02.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 -- 172.21.15.64:0/274798020 wait complete. 2024-04-29T20:23:02.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.022+0000 7ff9c49fe640 1 librados: shutdown 2024-04-29T20:23:02.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.7 2024-04-29T20:23:02.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.042+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0920003790 con 0x7f092c148890 2024-04-29T20:23:02.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.043+0000 7f0913fff640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f091806e2b0 0x7f0918070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.043+0000 7f092b7fe640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f091806e2b0 0x7f0918070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.043+0000 7f092b7fe640 10 monclient: get_auth_request con 0x7f091806e2b0 auth_method 0 2024-04-29T20:23:02.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.044+0000 7f092b7fe640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f091806e2b0 0x7f0918070770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f091c007b80 tx=0x7f091c001560 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.044+0000 7f0913fff640 10 client.4350.objecter ms_handle_connect 0x7f091806e2b0 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f092007a060 con 0x7f092c148890 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 10 client.4350.objecter ms_dispatch 0x7f092c000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 3 client.4350.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 3 client.4350.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 20 client.4350.objecter dump_active .. 0 homeless 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f092007c6c0 con 0x7f092c148890 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.059+0000 7f0913fff640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:02.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.086+0000 7f7a33df4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-29T20:23:02.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.086+0000 7f7a33df4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-29T20:23:02.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.086+0000 7f7a33df4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:02.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.086+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7a2c1407a0 con 0x7f7a2c0aa460 2024-04-29T20:23:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.132+0000 7f7a227fc640 1 -- 172.21.15.64:0/2530329078 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7a2805da90 con 0x7f7a2c0aa460 2024-04-29T20:23:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.132+0000 7f7a227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-29T20:23:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.132+0000 7f7a227fc640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:02.134 INFO:teuthology.orchestra.run.smithi064.stdout:94489280520 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 10 librados: watch_flush enter 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 10 librados: watch_flush exit 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 20 client.4265.objecter shutdown clearing up homeless session... 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 10 client.4265.objecter successfully canceled tick 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f7a1006e280 msgr2=0x7f7a10070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f7a1006e280 0x7f7a10070740 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c036000 tx=0x7f7a1c030000 comp rx=0 tx=0).stop 2024-04-29T20:23:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 10 monclient: shutdown 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 msgr2=0x7f7a2c143a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 0x7f7a2c143a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800b4a0 tx=0x7f7a2800b960 comp rx=0 tx=0).stop 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 shutdown_connections 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f7a1006e280 0x7f7a10070740 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b5c60 0x7f7a2c13b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a2c0b2230 0x7f7a2c1dbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 --2- 172.21.15.64:0/2530329078 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f7a2c0aa460 0x7f7a2c143a50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 >> 172.21.15.64:0/2530329078 conn(0x7f7a2c06c960 msgr2=0x7f7a2c0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 shutdown_connections 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 -- 172.21.15.64:0/2530329078 wait complete. 2024-04-29T20:23:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.134+0000 7f7a33df4640 1 librados: shutdown 2024-04-29T20:23:02.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280522 got 94489280520 for osd.3 2024-04-29T20:23:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.150+0000 7f3bc4ff9640 1 -- 172.21.15.64:0/2979995131 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3bc80adcf0 con 0x7f3bc80b2230 2024-04-29T20:23:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.150+0000 7f3bc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-29T20:23:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.150+0000 7f3bc4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:02.151 INFO:teuthology.orchestra.run.smithi064.stdout:94489280520 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.151+0000 7f3bce598640 10 librados: watch_flush enter 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 10 librados: watch_flush exit 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 20 client.4344.objecter shutdown clearing up homeless session... 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 10 client.4344.objecter successfully canceled tick 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3bb406e2d0 msgr2=0x7f3bb4070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3bb406e2d0 0x7f3bb4070790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8001560 tx=0x7f3bb8007990 comp rx=0 tx=0).stop 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 10 monclient: shutdown 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 msgr2=0x7f3bc80ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 0x7f3bc80ae3d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b440 tx=0x7f3bbc00b900 comp rx=0 tx=0).stop 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 shutdown_connections 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3bb406e2d0 0x7f3bb4070790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc80b5c60 0x7f3bc813bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bc80b2230 0x7f3bc80ae3d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 --2- 172.21.15.64:0/2979995131 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3bc80aa460 0x7f3bc8143760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 >> 172.21.15.64:0/2979995131 conn(0x7f3bc806c960 msgr2=0x7f3bc80b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 shutdown_connections 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 -- 172.21.15.64:0/2979995131 wait complete. 2024-04-29T20:23:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.152+0000 7f3bce598640 1 librados: shutdown 2024-04-29T20:23:02.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.154+0000 7f87b1ffb640 1 -- 172.21.15.64:0/2379787752 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f87b405e170 con 0x7f87bc0ad6a0 2024-04-29T20:23:02.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.154+0000 7f87b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-29T20:23:02.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.154+0000 7f87b1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:02.156 INFO:teuthology.orchestra.run.smithi064.stdout:94489280519 2024-04-29T20:23:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 10 librados: watch_flush enter 2024-04-29T20:23:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 10 librados: watch_flush exit 2024-04-29T20:23:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 20 client.4341.objecter shutdown clearing up homeless session... 2024-04-29T20:23:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 10 client.4341.objecter successfully canceled tick 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f87a006e280 msgr2=0x7f87a0070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f87a006e280 0x7f87a0070740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f87bc069370 tx=0x7f87b8006ba0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 10 monclient: shutdown 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 msgr2=0x7f87bc13c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc13c520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b9c0 tx=0x7f87b400be80 comp rx=0 tx=0).stop 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 shutdown_connections 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f87a006e280 0x7f87a0070740 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc14ba90 0x7f87bc1404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87bc0ad6a0 0x7f87bc13c520 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 --2- 172.21.15.64:0/2379787752 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f87bc0aad50 0x7f87bc13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 >> 172.21.15.64:0/2379787752 conn(0x7f87bc0b4b60 msgr2=0x7f87bc0ab8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 shutdown_connections 2024-04-29T20:23:02.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.157+0000 7f87c33f7640 1 -- 172.21.15.64:0/2379787752 wait complete. 2024-04-29T20:23:02.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.158+0000 7f87c33f7640 1 librados: shutdown 2024-04-29T20:23:02.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280522 got 94489280520 for osd.1 2024-04-29T20:23:02.175 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280521 got 94489280519 for osd.0 2024-04-29T20:23:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.270+0000 7f0931f63640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-29T20:23:02.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.270+0000 7f0931f63640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-29T20:23:02.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.270+0000 7f0931f63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:23:02.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.270+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f092c144c30 con 0x7f092c148890 2024-04-29T20:23:02.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.274+0000 7f0913fff640 1 -- 172.21.15.64:0/1203656479 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f092007bfc0 con 0x7f092c148890 2024-04-29T20:23:02.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.274+0000 7f0913fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-04-29T20:23:02.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.274+0000 7f0913fff640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:02.276 INFO:teuthology.orchestra.run.smithi064.stdout:85899345929 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 10 librados: watch_flush enter 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 10 librados: watch_flush exit 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 20 client.4350.objecter shutdown clearing up homeless session... 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 10 client.4350.objecter successfully canceled tick 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f091806e2b0 msgr2=0x7f0918070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f091806e2b0 0x7f0918070770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f091c007b80 tx=0x7f091c001560 comp rx=0 tx=0).stop 2024-04-29T20:23:02.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 10 monclient: shutdown 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 msgr2=0x7f092c1492f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 0x7f092c1492f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0920009980 tx=0x7f09200047d0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 shutdown_connections 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f091806e2b0 0x7f0918070770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f092c14acc0 0x7f092c139ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f092c148890 0x7f092c1492f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 --2- 172.21.15.64:0/1203656479 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f092c1484b0 0x7f092c13abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 >> 172.21.15.64:0/1203656479 conn(0x7f092c06c960 msgr2=0x7f092c13dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 shutdown_connections 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 -- 172.21.15.64:0/1203656479 wait complete. 2024-04-29T20:23:02.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.277+0000 7f0931f63640 1 librados: shutdown 2024-04-29T20:23:02.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.6 2024-04-29T20:23:02.290 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:02.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-29T20:23:02.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-29T20:23:02.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.585+0000 7fc158e10640 10 monclient: get_monmap_and_config 2024-04-29T20:23:02.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.585+0000 7fc158e10640 10 monclient: build_initial_monmap 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.585+0000 7fc158e10640 10 monclient: monmap: 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:02.586806+0000 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:02.586806+0000 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.586+0000 7fc158e10640 1 Processor -- start 2024-04-29T20:23:02.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.586+0000 7fc158e10640 1 -- start start 2024-04-29T20:23:02.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.586+0000 7fc158e10640 10 monclient: init 2024-04-29T20:23:02.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1540b2230 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1540b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): picked mon.noname-b con 0x7fc1540aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc1540b5c60 0x7fc15414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): picked mon.noname-a con 0x7fc1540b5c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1540ac1f0 con 0x7fc1540b2230 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fc1540ac4d0 con 0x7fc1540b5c60 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc1540b3440 con 0x7fc1540aa460 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.587+0000 7fc158e10640 10 monclient(hunting): authenticate will time out at 1531.746990s 2024-04-29T20:23:02.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc15259c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc1540b2230 0x7fc154146970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40644/0 (socket says 172.21.15.64:40644) 2024-04-29T20:23:02.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc152d9d640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc1540b5c60 0x7fc15414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc15259c640 1 -- 172.21.15.64:0/2824783360 learned_addr learned my addr 172.21.15.64:0/2824783360 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:02.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc151d9b640 1 --2- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1540b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc152d9d640 10 monclient(hunting): get_auth_request con 0x7fc1540b5c60 auth_method 0 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc152d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc152d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc152d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc151d9b640 10 monclient(hunting): get_auth_request con 0x7fc1540aa460 auth_method 0 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc151d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc151d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.588+0000 7fc151d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.589+0000 7fc152d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.589+0000 7fc152d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.589+0000 7fc152d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.589+0000 7fc151d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.589+0000 7fc151d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.589+0000 7fc151d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc15159a640 1 -- 172.21.15.64:0/2824783360 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 409977908 0 0) 0x7fc1540ac1f0 con 0x7fc1540b2230 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc15159a640 10 monclient(hunting): _negotiate 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc15159a640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc15159a640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc15159a640 10 monclient(hunting): my global_id is 4363 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc15159a640 1 -- 172.21.15.64:0/2824783360 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc140003520 con 0x7fc1540b2230 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-04-29T20:23:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 1 -- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 msgr2=0x7fc1540b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 1 --2- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1540b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 1 -- 172.21.15.64:0/2824783360 >> v1:172.21.15.167:6789/0 conn(0x7fc1540b2230 legacy=0x7fc154146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 1 monclient: found mon.noname-a 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.590+0000 7fc152d9d640 1 -- 172.21.15.64:0/2824783360 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1540aa8a0 con 0x7fc1540b5c60 2024-04-29T20:23:02.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.593+0000 7fc152d9d640 1 --2- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc15414dce0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc144009d90 tx=0x7fc14402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a25519e87af976 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 1 -- 172.21.15.64:0/2824783360 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144039070 con 0x7fc1540b5c60 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: dump: 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.599 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 1 -- 172.21.15.64:0/2824783360 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14402cbc0 con 0x7fc1540b5c60 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc158e10640 5 monclient: authenticate success, global_id 4356 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc158e10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc158e10640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc158e10640 4 set_mon_vals no callback set 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 1 -- 172.21.15.64:0/2824783360 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144034440 con 0x7fc1540b5c60 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.597+0000 7fc15159a640 10 monclient: dump: 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.601 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 10 monclient: shutdown 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 -- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 msgr2=0x7fc15414dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 --2- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc15414dce0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc144009d90 tx=0x7fc14402be10 comp rx=0 tx=0).stop 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 -- 172.21.15.64:0/2824783360 shutdown_connections 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 --2- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc15414dce0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 --2- 172.21.15.64:0/2824783360 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1540b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 -- 172.21.15.64:0/2824783360 >> 172.21.15.64:0/2824783360 conn(0x7fc15406c960 msgr2=0x7fc15406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 -- 172.21.15.64:0/2824783360 shutdown_connections 2024-04-29T20:23:02.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.598+0000 7fc158e10640 1 -- 172.21.15.64:0/2824783360 wait complete. 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.599+0000 7fc158e10640 10 monclient: build_initial_monmap 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.599+0000 7fc158e10640 10 monclient: monmap: 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 1 librados: starting msgr at 2024-04-29T20:23:02.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 1 librados: starting objecter 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 1 Processor -- start 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 1 -- start start 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 1 librados: setting wanted keys 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 1 librados: calling monclient init 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.600+0000 7fc158e10640 10 monclient: init 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1541403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): picked mon.noname-c con 0x7fc1540aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1540b2230 0x7fc154143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): picked mon.noname-b con 0x7fc1540b2230 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc1540ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): picked mon.noname-a con 0x7fc1540b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc1540b2790 con 0x7fc1540b5c60 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc1540b2610 con 0x7fc1540b2230 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc15414e220 con 0x7fc1540aa460 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc158e10640 10 monclient(hunting): authenticate will time out at 1531.760635s 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc152d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc1540ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc151d9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1540b2230 0x7fc154143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.601+0000 7fc15259c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1541403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc1540ac980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34658/0 (socket says 172.21.15.64:34658) 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 1 -- 172.21.15.64:0/2856876108 learned_addr learned my addr 172.21.15.64:0/2856876108 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): get_auth_request con 0x7fc1540b5c60 auth_method 0 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): get_auth_request con 0x7fc1540b2230 auth_method 0 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc15259c640 10 monclient(hunting): get_auth_request con 0x7fc1540aa460 auth_method 0 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc15259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc15259c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc15259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc151d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.602+0000 7fc152d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.603+0000 7fc15259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.603+0000 7fc15259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.603+0000 7fc15259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-04-29T20:23:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 1 -- 172.21.15.64:0/2856876108 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 msgr2=0x7fc1541403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1541403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 1 -- 172.21.15.64:0/2856876108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 msgr2=0x7fc1540ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc1540ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 1 monclient: found mon.noname-b 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.605+0000 7fc151d9b640 1 -- 172.21.15.64:0/2856876108 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1540adb70 con 0x7fc1540b2230 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.607+0000 7fc151d9b640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1540b2230 0x7fc154143f40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00e850 tx=0x7fc13c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.607+0000 7fc14affd640 10 client.?.objecter ms_handle_connect 0x7fc1540b2230 2024-04-29T20:23:02.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.607+0000 7fc14affd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c017070 con 0x7fc1540b2230 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: dump: 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.614 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc13c00c940 con 0x7fc1540b2230 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 5 monclient: authenticate success, global_id 4366 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 10 monclient: _renew_subs 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1540b15b0 con 0x7fc1540b2230 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00cd10 con 0x7fc1540b2230 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14affd640 10 monclient: dump: 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc14bfff640 4 set_mon_vals no callback set 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 10 monclient: _renew_subs 2024-04-29T20:23:02.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1541d8740 con 0x7fc1540b2230 2024-04-29T20:23:02.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.611+0000 7fc158e10640 1 librados: init done 2024-04-29T20:23:02.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.612+0000 7fc158e10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.612+0000 7fc158e10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.612+0000 7fc158e10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.612+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15404fce0 con 0x7fc1540b2230 2024-04-29T20:23:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.629+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc13c01a070 con 0x7fc1540b2230 2024-04-29T20:23:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.630+0000 7fc14affd640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc13406e300 0x7fc1340707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.630+0000 7fc15259c640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc13406e300 0x7fc1340707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.630+0000 7fc15259c640 10 monclient: get_auth_request con 0x7fc13406e300 auth_method 0 2024-04-29T20:23:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.631+0000 7fc15259c640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc13406e300 0x7fc1340707c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc14c005880 tx=0x7fc14c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.631+0000 7fc14affd640 10 client.4366.objecter ms_handle_connect 0x7fc13406e300 2024-04-29T20:23:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.633+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7fc13c058050 con 0x7fc1540b2230 2024-04-29T20:23:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.633+0000 7fc14affd640 10 client.4366.objecter ms_dispatch 0x7fc154000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.633+0000 7fc14affd640 3 client.4366.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.633+0000 7fc14affd640 3 client.4366.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.633+0000 7fc14affd640 20 client.4366.objecter dump_active .. 0 homeless 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.641+0000 7fa962d06640 10 monclient: get_monmap_and_config 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.641+0000 7fa962d06640 10 monclient: build_initial_monmap 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.641+0000 7fa962d06640 10 monclient: monmap: 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:02.641927+0000 2024-04-29T20:23:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:02.641927+0000 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:02.643 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.641+0000 7fa962d06640 1 Processor -- start 2024-04-29T20:23:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.641+0000 7fa962d06640 1 -- start start 2024-04-29T20:23:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.641+0000 7fa962d06640 10 monclient: init 2024-04-29T20:23:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa95c0b2230 0x7fa95c146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient(hunting): picked mon.noname-c con 0x7fa95c0b2230 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa95c0aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa95c0b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95c146eb0 con 0x7fa95c0b5c60 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95c0ac1f0 con 0x7fa95c0aa460 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fa95c0ac4d0 con 0x7fa95c0b2230 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.642+0000 7fa962d06640 10 monclient(hunting): authenticate will time out at 1531.802045s 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.643+0000 7fa9612a3640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa95c0b5c60 0x7fa95c14dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51912/0 (socket says 172.21.15.64:51912) 2024-04-29T20:23:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.643+0000 7fa9612a3640 1 -- 172.21.15.64:0/1549490416 learned_addr learned my addr 172.21.15.64:0/1549490416 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.645+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 197278521 0 0) 0x7fa95c146eb0 con 0x7fa95c0b5c60 2024-04-29T20:23:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.645+0000 7fa95b7fe640 10 monclient(hunting): _negotiate 2024-04-29T20:23:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.645+0000 7fa95b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.645+0000 7fa95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.645+0000 7fa95b7fe640 10 monclient(hunting): my global_id is 4362 2024-04-29T20:23:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.645+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa948003520 con 0x7fa95c0b5c60 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3593729697 0 0) 0x7fa948003520 con 0x7fa95c0b5c60 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 >> v2:172.21.15.167:3300/0 conn(0x7fa95c0b2230 msgr2=0x7fa95c146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 --2- 172.21.15.64:0/1549490416 >> v2:172.21.15.167:3300/0 conn(0x7fa95c0b2230 0x7fa95c146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 >> v1:172.21.15.167:6789/0 conn(0x7fa95c0aa460 legacy=0x7fa95c0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 monclient: found mon.noname-a-legacy 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.64:6789/0 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c0aa8d0 con 0x7fa95c0b5c60 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 525552804 0 0) 0x7fa944001670 con 0x7fa95c0b5c60 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: dump: 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.652 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] but i'm connected to v1:172.21.15.64:6789/0, reconnecting 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 >> v1:172.21.15.64:6789/0 conn(0x7fa95c0b5c60 legacy=0x7fa95c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa948003a00 0x7fa948003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): picked mon.b con 0x7fa948003a00 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa948004800 0x7fa948004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): picked mon.a con 0x7fa948004800 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa94800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): picked mon.c con 0x7fa9480052e0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c0ac4d0 con 0x7fa948004800 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa94800c230 con 0x7fa948003a00 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa94800c430 con 0x7fa9480052e0 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.649+0000 7fa95b7fe640 10 monclient(hunting): _finish_auth 0 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.650+0000 7fa9612a3640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa94800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.650+0000 7fa9612a3640 10 monclient(hunting): get_auth_request con 0x7fa9480052e0 auth_method 0 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.650+0000 7fa9612a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.650+0000 7fa9612a3640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.650+0000 7fa9612a3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.651+0000 7fa9612a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.651+0000 7fa9612a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.651+0000 7fa9612a3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.652+0000 7fa95bfff640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa948004800 0x7fa948004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa95bfff640 10 monclient(hunting): get_auth_request con 0x7fa948004800 auth_method 0 2024-04-29T20:23:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa95bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa95bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa95bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 10 monclient(hunting): handle_auth_done global_id 4362 payload 293 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 1 -- 172.21.15.64:0/1549490416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa948003a00 msgr2=0x7fa948003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa948003a00 0x7fa948003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 1 -- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa948004800 msgr2=0x7fa948004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa948004800 0x7fa948004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 1 monclient: found mon.c 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 1 -- 172.21.15.64:0/1549490416 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa948003520 con 0x7fa9480052e0 2024-04-29T20:23:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa9612a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa962d06640 5 monclient: authenticate success, global_id 4362 2024-04-29T20:23:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.653+0000 7fa962d06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.654+0000 7fa9612a3640 1 -- 172.21.15.64:0/1549490416 reap_dead start 2024-04-29T20:23:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.655+0000 7fa9612a3640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa94800b9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa944001eb0 tx=0x7fa948005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a9d184654d6dde5 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944007c80 con 0x7fa9480052e0 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: dump: 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.660 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa944030070 con 0x7fa9480052e0 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 4 set_mon_vals no callback set 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 1 -- 172.21.15.64:0/1549490416 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944005d30 con 0x7fa9480052e0 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa95b7fe640 10 monclient: dump: 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.662 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 10 monclient: shutdown 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 -- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 msgr2=0x7fa94800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa94800b9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa944001eb0 tx=0x7fa948005830 comp rx=0 tx=0).stop 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 -- 172.21.15.64:0/1549490416 shutdown_connections 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 --2- 172.21.15.64:0/1549490416 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa94800b9c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 -- 172.21.15.64:0/1549490416 >> 172.21.15.64:0/1549490416 conn(0x7fa95c06c960 msgr2=0x7fa95c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 -- 172.21.15.64:0/1549490416 shutdown_connections 2024-04-29T20:23:02.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.659+0000 7fa962d06640 1 -- 172.21.15.64:0/1549490416 wait complete. 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 10 monclient: build_initial_monmap 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 10 monclient: monmap: 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 1 librados: starting msgr at 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 1 librados: starting objecter 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 1 Processor -- start 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 1 -- start start 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 1 librados: setting wanted keys 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 1 librados: calling monclient init 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.660+0000 7fa962d06640 10 monclient: init 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa95c145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): picked mon.noname-c con 0x7fa9480052e0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa95c0ac710 0x7fa95c0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): picked mon.noname-a con 0x7fa95c0ac710 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa95c1dc360 0x7fa95c0acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): picked mon.noname-b con 0x7fa95c1dc360 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c0b6210 con 0x7fa95c0ac710 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c0b6090 con 0x7fa95c1dc360 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa95c146970 con 0x7fa9480052e0 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.661+0000 7fa962d06640 10 monclient(hunting): authenticate will time out at 1531.821223s 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa95c145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa95bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa95c0ac710 0x7fa95c0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa95c145180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36842/0 (socket says 172.21.15.64:36842) 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 1 -- 172.21.15.64:0/3191513183 learned_addr learned my addr 172.21.15.64:0/3191513183 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 10 monclient(hunting): get_auth_request con 0x7fa9480052e0 auth_method 0 2024-04-29T20:23:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa960aa2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa95bfff640 10 monclient(hunting): get_auth_request con 0x7fa95c0ac710 auth_method 0 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa95bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa95bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.662+0000 7fa95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.663+0000 7fa960aa2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.663+0000 7fa960aa2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.663+0000 7fa960aa2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.663+0000 7fa95bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.663+0000 7fa95bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.663+0000 7fa95bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-04-29T20:23:02.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 1 -- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa95c1dc360 msgr2=0x7fa95c0acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa95c1dc360 0x7fa95c0acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 1 -- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa95c0ac710 msgr2=0x7fa95c0b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa95c0ac710 0x7fa95c0b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 1 monclient: found mon.noname-c 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.666+0000 7fa960aa2640 1 -- 172.21.15.64:0/3191513183 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c0b05d0 con 0x7fa9480052e0 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.668+0000 7fa960aa2640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa95c145180 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa95400b9a0 tx=0x7fa95400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.668+0000 7fa9597fa640 10 client.?.objecter ms_handle_connect 0x7fa9480052e0 2024-04-29T20:23:02.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.668+0000 7fa9597fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:02.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954019070 con 0x7fa9480052e0 2024-04-29T20:23:02.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: dump: 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa95400cb40 con 0x7fa9480052e0 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954014440 con 0x7fa9480052e0 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa9597fa640 10 monclient: dump: 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.676 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 5 monclient: authenticate success, global_id 4277 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 10 monclient: _renew_subs 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95c0b1480 con 0x7fa9480052e0 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa95affd640 4 set_mon_vals no callback set 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 10 monclient: _renew_subs 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95c1dc9a0 con 0x7fa9480052e0 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.672+0000 7fa962d06640 1 librados: init done 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.673+0000 7fa962d06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.673+0000 7fa962d06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.673+0000 7fa962d06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:02.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.673+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa95c04fce0 con 0x7fa9480052e0 2024-04-29T20:23:02.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-29T20:23:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.893+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc13c059fb0 con 0x7fc1540b2230 2024-04-29T20:23:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.893+0000 7fc14affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.893+0000 7fc14affd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.931+0000 7ff1f40f2640 10 monclient: get_monmap_and_config 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.931+0000 7ff1f40f2640 10 monclient: build_initial_monmap 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.931+0000 7ff1f40f2640 10 monclient: monmap: 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:02.932662+0000 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:02.932662+0000 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:02.934 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:02.934 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:02.934 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:02.934 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:02.934 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.932+0000 7ff1f40f2640 1 Processor -- start 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.932+0000 7ff1f40f2640 1 -- start start 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.932+0000 7ff1f40f2640 10 monclient: init 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1ec0b2230 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1ec0aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff1ec0b5c60 0x7ff1ec14ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ec0b5c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec0ac1f0 con 0x7ff1ec0b2230 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec0ac4d0 con 0x7ff1ec0aa460 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ec0aa8d0 con 0x7ff1ec0b5c60 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f40f2640 10 monclient(hunting): authenticate will time out at 1532.092903s 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f268f640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff1ec0b5c60 0x7ff1ec14ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f168d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7ff1ec0aa460 0x7ff1ec0b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40656/0 (socket says 172.21.15.64:40656) 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.933+0000 7ff1f168d640 1 -- 172.21.15.64:0/1393714594 learned_addr learned my addr 172.21.15.64:0/1393714594 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.934+0000 7ff1f268f640 10 monclient(hunting): get_auth_request con 0x7ff1ec0b5c60 auth_method 0 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.934+0000 7ff1f268f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.934+0000 7ff1f268f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.934+0000 7ff1f268f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.935+0000 7ff1f268f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.935+0000 7ff1f268f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.935+0000 7ff1f268f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.936+0000 7ff1f0e8c640 1 -- 172.21.15.64:0/1393714594 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640996685 0 0) 0x7ff1ec0ac4d0 con 0x7ff1ec0aa460 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.936+0000 7ff1f0e8c640 10 monclient(hunting): _negotiate 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.936+0000 7ff1f0e8c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.936+0000 7ff1f0e8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.936+0000 7ff1f0e8c640 10 monclient(hunting): my global_id is 4369 2024-04-29T20:23:02.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.936+0000 7ff1f0e8c640 1 -- 172.21.15.64:0/1393714594 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d8003520 con 0x7ff1ec0aa460 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 1 -- 172.21.15.64:0/1393714594 >> v1:172.21.15.167:6789/0 conn(0x7ff1ec0aa460 legacy=0x7ff1ec0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 1 -- 172.21.15.64:0/1393714594 >> v1:172.21.15.64:6789/0 conn(0x7ff1ec0b2230 legacy=0x7ff1ec146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 1 monclient: found mon.noname-a 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.937+0000 7ff1f268f640 1 -- 172.21.15.64:0/1393714594 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec0b2610 con 0x7ff1ec0b5c60 2024-04-29T20:23:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.941+0000 7ff1f268f640 1 --2- 172.21.15.64:0/1393714594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0b5c60 0x7ff1ec14ecf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009d90 tx=0x7ff1e402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608fb47672e43766 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 1 -- 172.21.15.64:0/1393714594 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4039070 con 0x7ff1ec0b5c60 2024-04-29T20:23:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: dump: 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 1 -- 172.21.15.64:0/1393714594 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e402cd00 con 0x7ff1ec0b5c60 2024-04-29T20:23:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 1 -- 172.21.15.64:0/1393714594 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4034660 con 0x7ff1ec0b5c60 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f0e8c640 10 monclient: dump: 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.949 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f40f2640 5 monclient: authenticate success, global_id 4368 2024-04-29T20:23:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f40f2640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.946+0000 7ff1f40f2640 4 set_mon_vals no callback set 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 10 monclient: shutdown 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 -- 172.21.15.64:0/1393714594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0b5c60 msgr2=0x7ff1ec14ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/1393714594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0b5c60 0x7ff1ec14ecf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009d90 tx=0x7ff1e402bf50 comp rx=0 tx=0).stop 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 -- 172.21.15.64:0/1393714594 shutdown_connections 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/1393714594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0b5c60 0x7ff1ec14ecf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 -- 172.21.15.64:0/1393714594 >> 172.21.15.64:0/1393714594 conn(0x7ff1ec06c960 msgr2=0x7ff1ec06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 -- 172.21.15.64:0/1393714594 shutdown_connections 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.947+0000 7ff1f40f2640 1 -- 172.21.15.64:0/1393714594 wait complete. 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 10 monclient: build_initial_monmap 2024-04-29T20:23:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 10 monclient: monmap: 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 1 librados: starting msgr at 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 1 librados: starting objecter 2024-04-29T20:23:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 1 Processor -- start 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 1 -- start start 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 1 librados: setting wanted keys 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 1 librados: calling monclient init 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.948+0000 7ff1f40f2640 10 monclient: init 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0aa460 0x7ff1ec0b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ec0aa460 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 0x7ff1ec1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ec0b2230 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1ec0b5c60 0x7ff1ec1d8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ec0b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec14f3b0 con 0x7ff1ec0aa460 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec14f230 con 0x7ff1ec0b2230 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ec14f530 con 0x7ff1ec0b5c60 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f40f2640 10 monclient(hunting): authenticate will time out at 1532.108831s 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f1e8e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0aa460 0x7ff1ec0b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.949+0000 7ff1f168d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 0x7ff1ec1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 0x7ff1ec1433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33768/0 (socket says 172.21.15.64:33768) 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 1 -- 172.21.15.64:0/3297743288 learned_addr learned my addr 172.21.15.64:0/3297743288 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): get_auth_request con 0x7ff1ec0b2230 auth_method 0 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): get_auth_request con 0x7ff1ec0aa460 auth_method 0 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f168d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.950+0000 7ff1f1e8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-04-29T20:23:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 1 -- 172.21.15.64:0/3297743288 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1ec0b5c60 msgr2=0x7ff1ec1d8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1ec0b5c60 0x7ff1ec1d8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 1 -- 172.21.15.64:0/3297743288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0aa460 msgr2=0x7ff1ec0b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0aa460 0x7ff1ec0b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 10 monclient: _finish_hunting 0 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 1 monclient: found mon.noname-b 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.953+0000 7ff1f168d640 1 -- 172.21.15.64:0/3297743288 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec13e470 con 0x7ff1ec0b2230 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.955+0000 7ff1f168d640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 0x7ff1ec1433e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00c530 tx=0x7ff1dc00c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.955+0000 7ff1e27fc640 10 client.?.objecter ms_handle_connect 0x7ff1ec0b2230 2024-04-29T20:23:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.955+0000 7ff1e27fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc017070 con 0x7ff1ec0b2230 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: dump: 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: _finish_auth 0 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: _check_auth_tickets 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1dc005ce0 con 0x7ff1ec0b2230 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0072b0 con 0x7ff1ec0b2230 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e27fc640 10 monclient: dump: 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 5 monclient: authenticate success, global_id 4372 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 10 monclient: _renew_subs 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1e3fff640 4 set_mon_vals no callback set 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec1d9350 con 0x7ff1ec0b2230 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 10 monclient: _renew_subs 2024-04-29T20:23:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec13cff0 con 0x7ff1ec0b2230 2024-04-29T20:23:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.959+0000 7ff1f40f2640 1 librados: init done 2024-04-29T20:23:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.960+0000 7ff1f40f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.960+0000 7ff1f40f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.960+0000 7ff1f40f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.960+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ec04fce0 con 0x7ff1ec0b2230 2024-04-29T20:23:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.968+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa95401c070 con 0x7fa9480052e0 2024-04-29T20:23:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.968+0000 7fa9597fa640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa93c06e2d0 0x7fa93c070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.968+0000 7fa95bfff640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa93c06e2d0 0x7fa93c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.968+0000 7fa95bfff640 10 monclient: get_auth_request con 0x7fa93c06e2d0 auth_method 0 2024-04-29T20:23:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.969+0000 7fa9597fa640 10 client.4277.objecter ms_handle_connect 0x7fa93c06e2d0 2024-04-29T20:23:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.969+0000 7fa95bfff640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa93c06e2d0 0x7fa93c070790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa94c007fd0 tx=0x7fa94c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.978+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1dc01a070 con 0x7ff1ec0b2230 2024-04-29T20:23:02.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.978+0000 7ff1e27fc640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff1d006e2d0 0x7ff1d0070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:02.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.978+0000 7ff1f1e8e640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff1d006e2d0 0x7ff1d0070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:02.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.978+0000 7ff1f1e8e640 10 monclient: get_auth_request con 0x7ff1d006e2d0 auth_method 0 2024-04-29T20:23:02.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.979+0000 7ff1e27fc640 10 client.4372.objecter ms_handle_connect 0x7ff1d006e2d0 2024-04-29T20:23:02.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.979+0000 7ff1f1e8e640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff1d006e2d0 0x7ff1d0070790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8001780 tx=0x7ff1e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:02.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.981+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7ff1dc057280 con 0x7ff1ec0b2230 2024-04-29T20:23:02.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.981+0000 7ff1e27fc640 10 client.4372.objecter ms_dispatch 0x7ff1ec000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:02.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.981+0000 7ff1e27fc640 3 client.4372.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:02.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.981+0000 7ff1e27fc640 3 client.4372.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:02.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.981+0000 7ff1e27fc640 20 client.4372.objecter dump_active .. 0 homeless 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7fa954057a00 con 0x7fa9480052e0 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 10 client.4277.objecter ms_dispatch 0x7fa95c000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 3 client.4277.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 3 client.4277.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 20 client.4277.objecter dump_active .. 0 homeless 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa954023020 con 0x7fa9480052e0 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:02.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:02.984+0000 7fa9597fa640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:03.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-29T20:23:03.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.093+0000 7fc158e10640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-29T20:23:03.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.093+0000 7fc158e10640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-29T20:23:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.093+0000 7fc158e10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.093+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc154071f90 con 0x7fc1540b2230 2024-04-29T20:23:03.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.138+0000 7fd91a9b5640 10 monclient: get_monmap_and_config 2024-04-29T20:23:03.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.138+0000 7fd91a9b5640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.138+0000 7fd91a9b5640 10 monclient: monmap: 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:03.139077+0000 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:03.139077+0000 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:03.140 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.138+0000 7fd91a9b5640 1 Processor -- start 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 1 -- start start 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient: init 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd9140b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9140aa460 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fd9140b5c60 0x7fd91414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd9140b5c60 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd914146eb0 con 0x7fd9140aa460 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fd9140ac1f0 con 0x7fd9140b5c60 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fd9140ac4d0 con 0x7fd9140b2230 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.139+0000 7fd91a9b5640 10 monclient(hunting): authenticate will time out at 1532.299192s 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd9137fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd9140aa460 0x7fd9140b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51930/0 (socket says 172.21.15.64:51930) 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd9137fe640 1 -- 172.21.15.64:0/1764560001 learned_addr learned my addr 172.21.15.64:0/1764560001 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd913fff640 1 --2- 172.21.15.64:0/1764560001 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd913fff640 10 monclient(hunting): get_auth_request con 0x7fd9140b2230 auth_method 0 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd913fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.140+0000 7fd913fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.141+0000 7fd913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.141+0000 7fd913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.141+0000 7fd913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.142+0000 7fd912ffd640 1 -- 172.21.15.64:0/1764560001 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2173035096 0 0) 0x7fd914146eb0 con 0x7fd9140aa460 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.143+0000 7fd912ffd640 10 monclient(hunting): _negotiate 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.143+0000 7fd912ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.143+0000 7fd912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.143+0000 7fd912ffd640 10 monclient(hunting): my global_id is 4374 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.143+0000 7fd912ffd640 1 -- 172.21.15.64:0/1764560001 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8fc003520 con 0x7fd9140aa460 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-04-29T20:23:03.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 1 -- 172.21.15.64:0/1764560001 >> v2:172.21.15.167:3300/0 conn(0x7fd9140b5c60 msgr2=0x7fd91414dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 1 --2- 172.21.15.64:0/1764560001 >> v2:172.21.15.167:3300/0 conn(0x7fd9140b5c60 0x7fd91414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 1 -- 172.21.15.64:0/1764560001 >> v1:172.21.15.64:6789/0 conn(0x7fd9140aa460 legacy=0x7fd9140b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 1 monclient: found mon.noname-b 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.144+0000 7fd913fff640 1 -- 172.21.15.64:0/1764560001 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9140aa8a0 con 0x7fd9140b2230 2024-04-29T20:23:03.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.146+0000 7fd913fff640 1 --2- 172.21.15.64:0/1764560001 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914146970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9080097a0 tx=0x7fd90802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19abb67049feaa01 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-29T20:23:03.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 1 -- 172.21.15.64:0/1764560001 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908035020 con 0x7fd9140b2230 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: dump: 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.154 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.154 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 1 -- 172.21.15.64:0/1764560001 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd908031830 con 0x7fd9140b2230 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 1 -- 172.21.15.64:0/1764560001 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908031c00 con 0x7fd9140b2230 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd912ffd640 10 monclient: dump: 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd91a9b5640 5 monclient: authenticate success, global_id 4280 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd91a9b5640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.151+0000 7fd91a9b5640 4 set_mon_vals no callback set 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 10 monclient: shutdown 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1764560001 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 msgr2=0x7fd914146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1764560001 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914146970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9080097a0 tx=0x7fd90802b8d0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1764560001 shutdown_connections 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1764560001 >> v2:172.21.15.167:3300/0 conn(0x7fd9140b5c60 0x7fd91414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1764560001 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914146970 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1764560001 >> 172.21.15.64:0/1764560001 conn(0x7fd91406c960 msgr2=0x7fd91406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1764560001 shutdown_connections 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.152+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1764560001 wait complete. 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 10 monclient: monmap: 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 1 librados: starting msgr at 2024-04-29T20:23:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 1 librados: starting objecter 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 1 Processor -- start 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 1 -- start start 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 1 librados: setting wanted keys 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 1 librados: calling monclient init 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.153+0000 7fd91a9b5640 10 monclient: init 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9140aa460 0x7fd9140acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd9140aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd9140b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9140b5c60 0x7fd9141d8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd9140b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd9140b2790 con 0x7fd9140b5c60 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd9140b2610 con 0x7fd9140aa460 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fd91414e220 con 0x7fd9140b2230 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.154+0000 7fd91a9b5640 10 monclient(hunting): authenticate will time out at 1532.314285s 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd918f52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9140b5c60 0x7fd9141d8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914141270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:36864/0 (socket says 172.21.15.64:36864) 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd918f52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9140b5c60 0x7fd9141d8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34710/0 (socket says 172.21.15.64:34710) 2024-04-29T20:23:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 1 -- 172.21.15.64:0/1094354763 learned_addr learned my addr 172.21.15.64:0/1094354763 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 10 monclient(hunting): get_auth_request con 0x7fd9140b2230 auth_method 0 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd9137fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd918f52640 10 monclient(hunting): get_auth_request con 0x7fd9140b5c60 auth_method 0 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd918f52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd918f52640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.155+0000 7fd918f52640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.156+0000 7fd918f52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.156+0000 7fd918f52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.156+0000 7fd918f52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.156+0000 7fd9137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.156+0000 7fd9137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.156+0000 7fd9137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-04-29T20:23:03.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 1 -- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9140aa460 msgr2=0x7fd9140acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9140aa460 0x7fd9140acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 1 -- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9140b5c60 msgr2=0x7fd9141d8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9140b5c60 0x7fd9141d8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 1 monclient: found mon.noname-c 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.160+0000 7fd9137fe640 1 -- 172.21.15.64:0/1094354763 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9140adef0 con 0x7fd9140b2230 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.163+0000 7fd9137fe640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914141270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd90400c460 tx=0x7fd90400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.163+0000 7fd910ff9640 10 client.?.objecter ms_handle_connect 0x7fd9140b2230 2024-04-29T20:23:03.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.163+0000 7fd910ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:03.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-29T20:23:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.167+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904017070 con 0x7fd9140b2230 2024-04-29T20:23:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.167+0000 7fd910ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.167+0000 7fd910ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.167+0000 7fd910ff9640 10 monclient: dump: 2024-04-29T20:23:03.169 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.167+0000 7fd910ff9640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd904005ce0 con 0x7fd9140b2230 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 5 monclient: authenticate success, global_id 4283 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 10 monclient: _renew_subs 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9141436d0 con 0x7fd9140b2230 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904007290 con 0x7fd9140b2230 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd910ff9640 10 monclient: dump: 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.171 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.172 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.172 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.172 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.172 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd9127fc640 4 set_mon_vals no callback set 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 10 monclient: _renew_subs 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9141d95d0 con 0x7fd9140b2230 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.168+0000 7fd91a9b5640 1 librados: init done 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.169+0000 7fd91a9b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.169+0000 7fd91a9b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.169+0000 7fd91a9b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.169+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd91404fce0 con 0x7fd9140b2230 2024-04-29T20:23:03.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-29T20:23:03.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.189+0000 7fa962d06640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-29T20:23:03.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.189+0000 7fa962d06640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-29T20:23:03.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.189+0000 7fa962d06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.189+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa95c0b75b0 con 0x7fa9480052e0 2024-04-29T20:23:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.203+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd90401a070 con 0x7fd9140b2230 2024-04-29T20:23:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.203+0000 7fd910ff9640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fd90006e280 0x7fd900070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.204+0000 7fd913fff640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fd90006e280 0x7fd900070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.204+0000 7fd913fff640 10 monclient: get_auth_request con 0x7fd90006e280 auth_method 0 2024-04-29T20:23:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.204+0000 7fd910ff9640 10 client.4283.objecter ms_handle_connect 0x7fd90006e280 2024-04-29T20:23:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.205+0000 7fd913fff640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fd90006e280 0x7fd900070740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd908036000 tx=0x7fd908030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.210+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7fd904056ff0 con 0x7fd9140b2230 2024-04-29T20:23:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.210+0000 7fd910ff9640 10 client.4283.objecter ms_dispatch 0x7fd914000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.210+0000 7fd910ff9640 3 client.4283.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.210+0000 7fd910ff9640 3 client.4283.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.210+0000 7fd910ff9640 20 client.4283.objecter dump_active .. 0 homeless 2024-04-29T20:23:03.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.232+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1dc021020 con 0x7ff1ec0b2230 2024-04-29T20:23:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.232+0000 7ff1e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.232+0000 7ff1e27fc640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:03.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.242+0000 7fc14affd640 1 -- 172.21.15.64:0/2856876108 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc13c05e2b0 con 0x7fc1540b2230 2024-04-29T20:23:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.242+0000 7fc14affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-29T20:23:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.242+0000 7fc14affd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:03.244 INFO:teuthology.orchestra.run.smithi064.stdout:64424509447 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 10 librados: watch_flush enter 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 10 librados: watch_flush exit 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 20 client.4366.objecter shutdown clearing up homeless session... 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 10 client.4366.objecter successfully canceled tick 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc13406e300 msgr2=0x7fc1340707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc13406e300 0x7fc1340707c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc14c005880 tx=0x7fc14c009040 comp rx=0 tx=0).stop 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 10 monclient: shutdown 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1540b2230 msgr2=0x7fc154143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1540b2230 0x7fc154143f40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00e850 tx=0x7fc13c00ed10 comp rx=0 tx=0).stop 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 shutdown_connections 2024-04-29T20:23:03.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc13406e300 0x7fc1340707c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1540b5c60 0x7fc1540ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1540b2230 0x7fc154143f40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 --2- 172.21.15.64:0/2856876108 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc1540aa460 0x7fc1541403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.245+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 >> 172.21.15.64:0/2856876108 conn(0x7fc15406c960 msgr2=0x7fc154147170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.246+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 shutdown_connections 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.246+0000 7fc158e10640 1 -- 172.21.15.64:0/2856876108 wait complete. 2024-04-29T20:23:03.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.246+0000 7fc158e10640 1 librados: shutdown 2024-04-29T20:23:03.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.2 2024-04-29T20:23:03.265 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:03.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.278+0000 7fb9c0524640 10 monclient: get_monmap_and_config 2024-04-29T20:23:03.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.278+0000 7fb9c0524640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.278+0000 7fb9c0524640 10 monclient: monmap: 2024-04-29T20:23:03.280 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.280 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.280 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:03.279631+0000 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:03.279631+0000 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:03.281 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.279+0000 7fb9c0524640 1 Processor -- start 2024-04-29T20:23:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.279+0000 7fb9c0524640 1 -- start start 2024-04-29T20:23:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.279+0000 7fb9c0524640 10 monclient: init 2024-04-29T20:23:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-29T20:23:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9b80b2230 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80aa460 0x7fb9b80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b80aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fb9b80b5c60 0x7fb9b814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b80b5c60 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b80ac1f0 con 0x7fb9b80b2230 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fb9b80ac4d0 con 0x7fb9b80b5c60 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b80b3440 con 0x7fb9b80aa460 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.281+0000 7fb9c0524640 10 monclient(hunting): authenticate will time out at 1532.441494s 2024-04-29T20:23:03.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.304+0000 7fb9be2c0640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb9b80b2230 0x7fb9b8146970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40660/0 (socket says 172.21.15.64:40660) 2024-04-29T20:23:03.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.304+0000 7fb9be2c0640 1 -- 172.21.15.64:0/1623443062 learned_addr learned my addr 172.21.15.64:0/1623443062 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:03.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.310+0000 7fa222311640 10 monclient: get_monmap_and_config 2024-04-29T20:23:03.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.310+0000 7fa222311640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.310+0000 7fa222311640 10 monclient: monmap: 2024-04-29T20:23:03.311 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:03.310999+0000 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:03.310999+0000 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:03.312 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.311+0000 7fa222311640 1 Processor -- start 2024-04-29T20:23:03.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.311+0000 7fa222311640 1 -- start start 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.311+0000 7fa222311640 10 monclient: init 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa21c0b2230 0x7fa21c146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient(hunting): picked mon.noname-a con 0x7fa21c0b2230 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa21c0aa460 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa21c0b5c60 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa21c146eb0 con 0x7fa21c0aa460 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa21c0ac1f0 con 0x7fa21c0b5c60 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fa21c0ac4d0 con 0x7fa21c0b2230 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.312+0000 7fa222311640 10 monclient(hunting): authenticate will time out at 1532.472039s 2024-04-29T20:23:03.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.313+0000 7fa2208ae640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fa21c0b5c60 0x7fa21c14dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40662/0 (socket says 172.21.15.64:40662) 2024-04-29T20:23:03.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.313+0000 7fa2208ae640 1 -- 172.21.15.64:0/427021107 learned_addr learned my addr 172.21.15.64:0/427021107 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.313+0000 7f86871e6640 10 monclient: get_monmap_and_config 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.313+0000 7f86871e6640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.313+0000 7f86871e6640 10 monclient: monmap: 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:03.314629+0000 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:03.314629+0000 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:03.317 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.314+0000 7f86871e6640 1 Processor -- start 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.314+0000 7f86871e6640 1 -- start start 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.314+0000 7f86871e6640 10 monclient: init 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86800b2230 0x7f8680146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient(hunting): picked mon.noname-b con 0x7f86800b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86800aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86800b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8680146eb0 con 0x7f86800b5c60 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86800ac1f0 con 0x7f86800aa460 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f86800ac4d0 con 0x7f86800b2230 2024-04-29T20:23:03.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.315+0000 7f86871e6640 10 monclient(hunting): authenticate will time out at 1532.474922s 2024-04-29T20:23:03.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.316+0000 7f867ffff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f86800aa460 0x7f86800b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:40676/0 (socket says 172.21.15.64:40676) 2024-04-29T20:23:03.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.316+0000 7f867ffff640 1 -- 172.21.15.64:0/482906331 learned_addr learned my addr 172.21.15.64:0/482906331 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:03.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.327+0000 7fb9beac1640 1 --2- 172.21.15.64:0/1623443062 >> v2:172.21.15.64:3300/0 conn(0x7fb9b80b5c60 0x7fb9b814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.328+0000 7fa21bfff640 1 --2- 172.21.15.64:0/427021107 >> v2:172.21.15.64:3300/0 conn(0x7fa21c0b2230 0x7fa21c146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.328+0000 7fb9beac1640 10 monclient(hunting): get_auth_request con 0x7fb9b80b5c60 auth_method 0 2024-04-29T20:23:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.328+0000 7fb9beac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.328+0000 7fb9beac1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.328+0000 7fb9beac1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fb9beac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fb9beac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fb9beac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fa21bfff640 10 monclient(hunting): get_auth_request con 0x7fa21c0b2230 auth_method 0 2024-04-29T20:23:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fa21bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.329+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fa21affd640 1 -- 172.21.15.64:0/427021107 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 233128706 0 0) 0x7fa21c0ac1f0 con 0x7fa21c0b5c60 2024-04-29T20:23:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fa21affd640 10 monclient(hunting): _negotiate 2024-04-29T20:23:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fa21affd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fa21affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fa21affd640 10 monclient(hunting): my global_id is 4378 2024-04-29T20:23:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fa21affd640 1 -- 172.21.15.64:0/427021107 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa204003520 con 0x7fa21c0b5c60 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fb9bd2be640 1 -- 172.21.15.64:0/1623443062 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175417220 0 0) 0x7fb9b80ac1f0 con 0x7fb9b80b2230 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fb9bd2be640 10 monclient(hunting): _negotiate 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fb9bd2be640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fb9bd2be640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fb9bd2be640 10 monclient(hunting): my global_id is 4375 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.332+0000 7fb9bd2be640 1 -- 172.21.15.64:0/1623443062 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a4003520 con 0x7fb9b80b2230 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.333+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1718865033 0 0) 0x7f86800ac1f0 con 0x7f86800aa460 2024-04-29T20:23:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.333+0000 7f867f7fe640 10 monclient(hunting): _negotiate 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.333+0000 7f867f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.333+0000 7f867f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.333+0000 7f867f7fe640 10 monclient(hunting): my global_id is 4381 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.333+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8670003520 con 0x7f86800aa460 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 1 -- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80aa460 msgr2=0x7fb9b80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 1 --2- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80aa460 0x7fb9b80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 1 -- 172.21.15.64:0/1623443062 >> v1:172.21.15.167:6789/0 conn(0x7fb9b80b2230 legacy=0x7fb9b8146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 1 monclient: found mon.noname-a 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.334+0000 7fb9beac1640 1 -- 172.21.15.64:0/1623443062 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b80aa8a0 con 0x7fb9b80b5c60 2024-04-29T20:23:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-04-29T20:23:03.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 1 -- 172.21.15.64:0/427021107 >> v1:172.21.15.167:6789/0 conn(0x7fa21c0b5c60 legacy=0x7fa21c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 1 -- 172.21.15.64:0/427021107 >> v1:172.21.15.64:6789/0 conn(0x7fa21c0aa460 legacy=0x7fa21c0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 1 monclient: found mon.noname-a 2024-04-29T20:23:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:23:03.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.337+0000 7fa21bfff640 1 -- 172.21.15.64:0/427021107 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21c0aa8d0 con 0x7fa21c0b2230 2024-04-29T20:23:03.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.339+0000 7fb9beac1640 1 --2- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80b5c60 0x7fb9b814dce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009d90 tx=0x7fb9b002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a48bb3550d2da94 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1325761698 0 0) 0x7f8670003520 con 0x7f86800aa460 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86800b2230 msgr2=0x7f8680146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86800b2230 0x7f8680146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 >> v1:172.21.15.64:6789/0 conn(0x7f86800b5c60 legacy=0x7f868014dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 monclient: found mon.noname-c-legacy 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.167:6789/0 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86800aa8a0 con 0x7f86800aa460 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 525552804 0 0) 0x7f866c001670 con 0x7f86800aa460 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f86871e6640 5 monclient: authenticate success, global_id 4381 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f86871e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: dump: 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] but i'm connected to v1:172.21.15.167:6789/0, reconnecting 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 >> v1:172.21.15.167:6789/0 conn(0x7f86800aa460 legacy=0x7f86800b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86700039b0 0x7f8670003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): picked mon.b con 0x7f86700039b0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86800b2230 0x7f8670004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): picked mon.c con 0x7f86800b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8670004e60 0x7f867000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): picked mon.a con 0x7f8670004e60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86800ac4d0 con 0x7f8670004e60 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86700054e0 con 0x7f86700039b0 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f867000bf00 con 0x7f86800b2230 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f867f7fe640 10 monclient(hunting): _finish_auth 0 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.345+0000 7f86871e6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.346+0000 7f8684f82640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86700039b0 0x7f8670003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.347+0000 7fa21bfff640 1 --2- 172.21.15.64:0/427021107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b2230 0x7fa21c146970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0097a0 tx=0x7fa20c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263b854e026c1d10 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): get_auth_request con 0x7f86700039b0 auth_method 0 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.349+0000 7f8684f82640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 10 monclient(hunting): handle_auth_done global_id 4381 payload 293 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 1 -- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86800b2230 msgr2=0x7f8670004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86800b2230 0x7f8670004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 1 -- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8670004e60 msgr2=0x7f867000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8670004e60 0x7f867000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 1 monclient: found mon.b 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 1 -- 172.21.15.64:0/482906331 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8670003520 con 0x7f86700039b0 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.351+0000 7f8684f82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.353+0000 7f8684f82640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86700039b0 0x7f8670003db0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8674009860 tx=0x7f8670005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=726f10cbe9b4fc98 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674037070 con 0x7f86700039b0 2024-04-29T20:23:03.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:03.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: dump: 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8674032620 con 0x7f86700039b0 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f86871e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f86871e6640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 1 -- 172.21.15.64:0/482906331 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674009460 con 0x7f86700039b0 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f86871e6640 4 set_mon_vals no callback set 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.357+0000 7f867f7fe640 10 monclient: dump: 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.361 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 10 monclient: shutdown 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 1 -- 172.21.15.64:0/482906331 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86700039b0 msgr2=0x7f8670003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 1 --2- 172.21.15.64:0/482906331 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86700039b0 0x7f8670003db0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8674009860 tx=0x7f8670005830 comp rx=0 tx=0).stop 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f8685783640 1 -- 172.21.15.64:0/482906331 reap_dead start 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 1 -- 172.21.15.64:0/482906331 shutdown_connections 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 1 -- 172.21.15.64:0/482906331 >> 172.21.15.64:0/482906331 conn(0x7f868006c960 msgr2=0x7f868006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 1 -- 172.21.15.64:0/482906331 shutdown_connections 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.358+0000 7f86871e6640 1 -- 172.21.15.64:0/482906331 wait complete. 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.359+0000 7f86871e6640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.359+0000 7f86871e6640 10 monclient: monmap: 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.359+0000 7f86871e6640 1 librados: starting msgr at 2024-04-29T20:23:03.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.359+0000 7f86871e6640 1 librados: starting objecter 2024-04-29T20:23:03.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7f86871e6640 1 Processor -- start 2024-04-29T20:23:03.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7f86871e6640 1 -- start start 2024-04-29T20:23:03.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7f86871e6640 1 librados: setting wanted keys 2024-04-29T20:23:03.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7f86871e6640 1 librados: calling monclient init 2024-04-29T20:23:03.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7f86871e6640 10 monclient: init 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 1 -- 172.21.15.64:0/1623443062 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0039070 con 0x7fb9b80b5c60 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: dump: 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.365 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 1 -- 172.21.15.64:0/1623443062 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b002cbc0 con 0x7fb9b80b5c60 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9c0524640 5 monclient: authenticate success, global_id 4380 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 1 -- 172.21.15.64:0/1623443062 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0034440 con 0x7fb9b80b5c60 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9bd2be640 10 monclient: dump: 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.366 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9c0524640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:03.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.360+0000 7fb9c0524640 4 set_mon_vals no callback set 2024-04-29T20:23:03.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:03.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86700039b0 0x7f86801433a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): picked mon.noname-c con 0x7f86700039b0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86801cdcb0 0x7f86801ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): picked mon.noname-a con 0x7f86801cdcb0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 0x7f86801cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 10 monclient: shutdown 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): picked mon.noname-b con 0x7f86801cee90 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86800b2790 con 0x7f86801cdcb0 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 -- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80b5c60 msgr2=0x7fb9b814dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 --2- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80b5c60 0x7fb9b814dce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009d90 tx=0x7fb9b002be10 comp rx=0 tx=0).stop 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86800b2610 con 0x7f86801cee90 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f8680146970 con 0x7f86700039b0 2024-04-29T20:23:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7f86871e6640 10 monclient(hunting): authenticate will time out at 1532.521160s 2024-04-29T20:23:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 -- 172.21.15.64:0/1623443062 shutdown_connections 2024-04-29T20:23:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 --2- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80b5c60 0x7fb9b814dce0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 --2- 172.21.15.64:0/1623443062 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80aa460 0x7fb9b80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 -- 172.21.15.64:0/1623443062 >> 172.21.15.64:0/1623443062 conn(0x7fb9b806c960 msgr2=0x7fb9b806cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 -- 172.21.15.64:0/1623443062 shutdown_connections 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.361+0000 7fb9c0524640 1 -- 172.21.15.64:0/1623443062 wait complete. 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 0x7f86801cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 0x7f86801cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33804/0 (socket says 172.21.15.64:33804) 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 1 -- 172.21.15.64:0/530002946 learned_addr learned my addr 172.21.15.64:0/530002946 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 10 monclient(hunting): get_auth_request con 0x7f86801cee90 auth_method 0 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7f8685783640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7fb9c0524640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.362+0000 7fb9c0524640 10 monclient: monmap: 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:03.374 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7f8685783640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7f8685783640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 1 librados: starting msgr at 2024-04-29T20:23:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 1 librados: starting objecter 2024-04-29T20:23:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7f8685783640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 1 Processor -- start 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 1 -- start start 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 1 librados: setting wanted keys 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 1 librados: calling monclient init 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.363+0000 7fb9c0524640 10 monclient: init 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80aa460 0x7fb9b8143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b80aa460 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 0x7fb9b81dc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b80b2230 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80b5c60 0x7fb9b8139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b80b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b80b2790 con 0x7fb9b80aa460 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b80b2610 con 0x7fb9b80b2230 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b814e220 con 0x7fb9b80b5c60 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.364+0000 7fb9c0524640 10 monclient(hunting): authenticate will time out at 1532.523977s 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 1 -- 172.21.15.64:0/530002946 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86700039b0 msgr2=0x7f86801433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86700039b0 0x7f86801433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 1 -- 172.21.15.64:0/530002946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86801cdcb0 msgr2=0x7f86801ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86801cdcb0 0x7f86801ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 1 monclient: found mon.noname-b 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.365+0000 7f8685783640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7f8685783640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7f8685783640 1 -- 172.21.15.64:0/530002946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86801d00a0 con 0x7f86801cee90 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 1 -- 172.21.15.64:0/427021107 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c034070 con 0x7fa21c0b2230 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: dump: 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 1 -- 172.21.15.64:0/427021107 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20c030610 con 0x7fa21c0b2230 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 1 -- 172.21.15.64:0/427021107 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0309e0 con 0x7fa21c0b2230 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa222311640 5 monclient: authenticate success, global_id 4383 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa21affd640 10 monclient: dump: 2024-04-29T20:23:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa222311640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.366+0000 7fa222311640 4 set_mon_vals no callback set 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 10 monclient: shutdown 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 -- 172.21.15.64:0/427021107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b2230 msgr2=0x7fa21c146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 --2- 172.21.15.64:0/427021107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b2230 0x7fa21c146970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0097a0 tx=0x7fa20c02b960 comp rx=0 tx=0).stop 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 -- 172.21.15.64:0/427021107 shutdown_connections 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 --2- 172.21.15.64:0/427021107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b2230 0x7fa21c146970 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 -- 172.21.15.64:0/427021107 >> 172.21.15.64:0/427021107 conn(0x7fa21c06c960 msgr2=0x7fa21c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 -- 172.21.15.64:0/427021107 shutdown_connections 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7fa222311640 1 -- 172.21.15.64:0/427021107 wait complete. 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7f8685783640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 0x7f86801cc770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f86800b5050 tx=0x7f867800c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7f867d7fa640 10 client.?.objecter ms_handle_connect 0x7f86801cee90 2024-04-29T20:23:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.367+0000 7f867d7fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.368+0000 7fa222311640 10 monclient: build_initial_monmap 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.368+0000 7fa222311640 10 monclient: monmap: 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:03.387 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.368+0000 7fa222311640 1 librados: starting msgr at 2024-04-29T20:23:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.368+0000 7fa222311640 1 librados: starting objecter 2024-04-29T20:23:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.368+0000 7fb9bdabf640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 0x7fb9b81dc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.368+0000 7fa222311640 1 Processor -- start 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.369+0000 7fa222311640 1 -- start start 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.369+0000 7fa222311640 1 librados: setting wanted keys 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.369+0000 7fa222311640 1 librados: calling monclient init 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.369+0000 7fa222311640 10 monclient: init 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 0x7fa21c0ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): picked mon.noname-b con 0x7fa21c0aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa21c0b2230 0x7fa21c13ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): picked mon.noname-c con 0x7fa21c0b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b5c60 0x7fa21c13b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): picked mon.noname-a con 0x7fa21c0b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c0b6210 con 0x7fa21c0b5c60 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c0b6090 con 0x7fa21c0aa460 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa21c14f400 con 0x7fa21c0b2230 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.370+0000 7fa222311640 10 monclient(hunting): authenticate will time out at 1532.529894s 2024-04-29T20:23:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.371+0000 7fb9bdabf640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 0x7fb9b81dc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33808/0 (socket says 172.21.15.64:33808) 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.371+0000 7fb9bdabf640 1 -- 172.21.15.64:0/2993955931 learned_addr learned my addr 172.21.15.64:0/2993955931 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.371+0000 7fb9bdabf640 10 monclient(hunting): get_auth_request con 0x7fb9b80b2230 auth_method 0 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.371+0000 7fb9bdabf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.371+0000 7fb9bdabf640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.371+0000 7fb9bdabf640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.372+0000 7fb9bdabf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.372+0000 7fb9bdabf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.372+0000 7fb9bdabf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678007110 con 0x7f86801cee90 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: dump: 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7fb9be2c0640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80aa460 0x7fb9b8143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86780072d0 con 0x7f86801cee90 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 5 monclient: authenticate success, global_id 4384 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 10 monclient: _renew_subs 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86801d0380 con 0x7f86801cee90 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86780076c0 con 0x7f86801cee90 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867d7fa640 10 monclient: dump: 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f867effd640 4 set_mon_vals no callback set 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 10 monclient: _renew_subs 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86801ce0d0 con 0x7f86801cee90 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.373+0000 7f86871e6640 1 librados: init done 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7f86871e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7f86871e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7f86871e6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868004fce0 con 0x7f86801cee90 2024-04-29T20:23:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7fb9be2c0640 10 monclient(hunting): get_auth_request con 0x7fb9b80aa460 auth_method 0 2024-04-29T20:23:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7fb9be2c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7fb9be2c0640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.374+0000 7fb9be2c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-04-29T20:23:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 1 -- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80b5c60 msgr2=0x7fb9b8139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80b5c60 0x7fb9b8139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 1 -- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80aa460 msgr2=0x7fb9b8143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80aa460 0x7fb9b8143e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 1 monclient: found mon.noname-b 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9bdabf640 1 -- 172.21.15.64:0/2993955931 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b81dbf30 con 0x7fb9b80b2230 2024-04-29T20:23:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.375+0000 7fb9be2c0640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80aa460 0x7fb9b8143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa2208ae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b5c60 0x7fa21c13b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 0x7fa21c0ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 0x7fa21c0ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:33810/0 (socket says 172.21.15.64:33810) 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 1 -- 172.21.15.64:0/3065622774 learned_addr learned my addr 172.21.15.64:0/3065622774 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 10 monclient(hunting): get_auth_request con 0x7fa21c0aa460 auth_method 0 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.376+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): get_auth_request con 0x7fa21c0b5c60 auth_method 0 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.378+0000 7fa2208ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.379+0000 7fb9bdabf640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 0x7fb9b81dc410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800b450 tx=0x7fb9a800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.379+0000 7fb9aeffd640 10 client.?.objecter ms_handle_connect 0x7fb9b80b2230 2024-04-29T20:23:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.379+0000 7fb9aeffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.381+0000 7fa21bfff640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.381+0000 7fa21bfff640 1 -- 172.21.15.64:0/3065622774 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa21c0b2230 msgr2=0x7fa21c13ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.381+0000 7fa21bfff640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa21c0b2230 0x7fa21c13ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.381+0000 7fa21bfff640 1 -- 172.21.15.64:0/3065622774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b5c60 msgr2=0x7fa21c13b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.382+0000 7fa21bfff640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b5c60 0x7fa21c13b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.382+0000 7fa21bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.382+0000 7fa21bfff640 1 monclient: found mon.noname-b 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.382+0000 7fa21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.382+0000 7fa21bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.382+0000 7fa21bfff640 1 -- 172.21.15.64:0/3065622774 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21c13b900 con 0x7fa21c0aa460 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.384+0000 7fa21bfff640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 0x7fa21c0ad7c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009890 tx=0x7fa20c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.384+0000 7fa218ff9640 10 client.?.objecter ms_handle_connect 0x7fa21c0aa460 2024-04-29T20:23:03.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.384+0000 7fa218ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:03.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.462+0000 7ff1f40f2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-29T20:23:03.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.462+0000 7ff1f40f2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-29T20:23:03.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.462+0000 7ff1f40f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.462+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff1ec071f90 con 0x7ff1ec0b2230 2024-04-29T20:23:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.507+0000 7fa9597fa640 1 -- 172.21.15.64:0/3191513183 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa954005e20 con 0x7fa9480052e0 2024-04-29T20:23:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.507+0000 7fa9597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-29T20:23:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.507+0000 7fa9597fa640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:03.509 INFO:teuthology.orchestra.run.smithi064.stdout:64424509449 2024-04-29T20:23:03.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 10 librados: watch_flush enter 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 10 librados: watch_flush exit 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 20 client.4277.objecter shutdown clearing up homeless session... 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 10 client.4277.objecter successfully canceled tick 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa93c06e2d0 msgr2=0x7fa93c070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa93c06e2d0 0x7fa93c070790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa94c007fd0 tx=0x7fa94c009040 comp rx=0 tx=0).stop 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 10 monclient: shutdown 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 msgr2=0x7fa95c145180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.509+0000 7fa962d06640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa95c145180 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa95400b9a0 tx=0x7fa95400be60 comp rx=0 tx=0).stop 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 shutdown_connections 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa93c06e2d0 0x7fa93c070790 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa95c1dc360 0x7fa95c0acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa95c0ac710 0x7fa95c0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 --2- 172.21.15.64:0/3191513183 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa9480052e0 0x7fa95c145180 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 >> 172.21.15.64:0/3191513183 conn(0x7fa95c06c960 msgr2=0x7fa95c0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 shutdown_connections 2024-04-29T20:23:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 -- 172.21.15.64:0/3191513183 wait complete. 2024-04-29T20:23:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.510+0000 7fa962d06640 1 librados: shutdown 2024-04-29T20:23:03.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.518+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd904021080 con 0x7fd9140b2230 2024-04-29T20:23:03.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.518+0000 7fd910ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.518+0000 7fd910ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:03.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.5 2024-04-29T20:23:03.522 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:03.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8015020 con 0x7fb9b80b2230 2024-04-29T20:23:03.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:03.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 10 monclient: dump: 2024-04-29T20:23:03.594 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.594 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 5 monclient: authenticate success, global_id 4387 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 10 monclient: _renew_subs 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b80ae130 con 0x7fb9b80b2230 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 10 monclient: _renew_subs 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b8141270 con 0x7fb9b80b2230 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.592+0000 7fb9c0524640 1 librados: init done 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.593+0000 7fb9c0524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.593+0000 7fb9c0524640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.593+0000 7fb9c0524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.593+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b804fce0 con 0x7fb9b80b2230 2024-04-29T20:23:03.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c034040 con 0x7fa21c0aa460 2024-04-29T20:23:03.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:03.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: dump: 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: _finish_auth 0 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:03.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20c039990 con 0x7fa21c0aa460 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 5 monclient: authenticate success, global_id 4390 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 10 monclient: _renew_subs 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21c0ad050 con 0x7fa21c0aa460 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c039d60 con 0x7fa21c0aa460 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:03.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa218ff9640 10 monclient: dump: 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa21a7fc640 4 set_mon_vals no callback set 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 10 monclient: _renew_subs 2024-04-29T20:23:03.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21c0adee0 con 0x7fa21c0aa460 2024-04-29T20:23:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.633+0000 7fa222311640 1 librados: init done 2024-04-29T20:23:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.634+0000 7fa222311640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.635+0000 7fa222311640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.635+0000 7fa222311640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.635+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa21c04fce0 con 0x7fa21c0aa460 2024-04-29T20:23:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a8011830 con 0x7fb9b80b2230 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9aeffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a80092b0 con 0x7fb9b80b2230 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9aeffd640 10 monclient: dump: 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:03.648 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.645+0000 7fb9bcabd640 4 set_mon_vals no callback set 2024-04-29T20:23:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.646+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8678007860 con 0x7f86801cee90 2024-04-29T20:23:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.646+0000 7f867d7fa640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f866406e300 0x7f86640707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.646+0000 7f8684f82640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f866406e300 0x7f86640707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.646+0000 7f8684f82640 10 monclient: get_auth_request con 0x7f866406e300 auth_method 0 2024-04-29T20:23:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.647+0000 7f8684f82640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f866406e300 0x7f86640707c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8674034000 tx=0x7f867402f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.647+0000 7f867d7fa640 10 client.4384.objecter ms_handle_connect 0x7f866406e300 2024-04-29T20:23:03.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.650+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f86780573d0 con 0x7f86801cee90 2024-04-29T20:23:03.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.650+0000 7f867d7fa640 10 client.4384.objecter ms_dispatch 0x7f8680000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:03.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.650+0000 7f867d7fa640 3 client.4384.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:03.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.650+0000 7f867d7fa640 3 client.4384.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:03.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.650+0000 7f867d7fa640 20 client.4384.objecter dump_active .. 0 homeless 2024-04-29T20:23:03.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.659+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8678059a30 con 0x7f86801cee90 2024-04-29T20:23:03.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.660+0000 7f867d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:03.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.660+0000 7f867d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.717+0000 7fd91a9b5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-29T20:23:03.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.717+0000 7fd91a9b5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-29T20:23:03.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.717+0000 7fd91a9b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:03.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.717+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd914071f90 con 0x7fd9140b2230 2024-04-29T20:23:03.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.723+0000 7fd910ff9640 1 -- 172.21.15.64:0/1094354763 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd904058f50 con 0x7fd9140b2230 2024-04-29T20:23:03.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.723+0000 7fd910ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-04-29T20:23:03.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.723+0000 7fd910ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:03.725 INFO:teuthology.orchestra.run.smithi064.stdout:64424509447 2024-04-29T20:23:03.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 10 librados: watch_flush enter 2024-04-29T20:23:03.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 10 librados: watch_flush exit 2024-04-29T20:23:03.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 20 client.4283.objecter shutdown clearing up homeless session... 2024-04-29T20:23:03.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 10 client.4283.objecter successfully canceled tick 2024-04-29T20:23:03.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fd90006e280 msgr2=0x7fd900070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fd90006e280 0x7fd900070740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd908036000 tx=0x7fd908030000 comp rx=0 tx=0).stop 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 10 monclient: shutdown 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 msgr2=0x7fd914141270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914141270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd90400c460 tx=0x7fd90400c920 comp rx=0 tx=0).stop 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 shutdown_connections 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fd90006e280 0x7fd900070740 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fd9140b2230 0x7fd914141270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9140b5c60 0x7fd9141d8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 --2- 172.21.15.64:0/1094354763 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9140aa460 0x7fd9140acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 >> 172.21.15.64:0/1094354763 conn(0x7fd91406c960 msgr2=0x7fd91406cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 shutdown_connections 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 -- 172.21.15.64:0/1094354763 wait complete. 2024-04-29T20:23:03.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.726+0000 7fd91a9b5640 1 librados: shutdown 2024-04-29T20:23:03.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.7 2024-04-29T20:23:03.739 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:03.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.849+0000 7f86871e6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-29T20:23:03.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.849+0000 7f86871e6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-29T20:23:03.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.849+0000 7f86871e6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:03.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:03.849+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f86800b7940 con 0x7f86801cee90 2024-04-29T20:23:04.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.153+0000 7ff1e27fc640 1 -- 172.21.15.64:0/3297743288 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff1dc0591e0 con 0x7ff1ec0b2230 2024-04-29T20:23:04.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.153+0000 7ff1e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-29T20:23:04.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.153+0000 7ff1e27fc640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:04.155 INFO:teuthology.orchestra.run.smithi064.stdout:64424509449 2024-04-29T20:23:04.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.154+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa20c037070 con 0x7fa21c0aa460 2024-04-29T20:23:04.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.154+0000 7fa218ff9640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa20806e280 0x7fa208070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.155+0000 7fa21b7fe640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa20806e280 0x7fa208070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.155+0000 7fa21b7fe640 10 monclient: get_auth_request con 0x7fa20806e280 auth_method 0 2024-04-29T20:23:04.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.155+0000 7fa21b7fe640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa20806e280 0x7fa208070740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa210000f60 tx=0x7fa2100079d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:04.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.155+0000 7fa218ff9640 10 client.4390.objecter ms_handle_connect 0x7fa20806e280 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 10 librados: watch_flush enter 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 10 librados: watch_flush exit 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 20 client.4372.objecter shutdown clearing up homeless session... 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 10 client.4372.objecter successfully canceled tick 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff1d006e2d0 msgr2=0x7ff1d0070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff1d006e2d0 0x7ff1d0070790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8001780 tx=0x7ff1e8009040 comp rx=0 tx=0).stop 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 10 monclient: shutdown 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:04.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 msgr2=0x7ff1ec1433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 0x7ff1ec1433e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00c530 tx=0x7ff1dc00c9f0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 shutdown_connections 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff1d006e2d0 0x7ff1d0070790 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff1ec0b5c60 0x7ff1ec1d8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1ec0b2230 0x7ff1ec1433e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 --2- 172.21.15.64:0/3297743288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1ec0aa460 0x7ff1ec0b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 >> 172.21.15.64:0/3297743288 conn(0x7ff1ec06c960 msgr2=0x7ff1ec0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 shutdown_connections 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 -- 172.21.15.64:0/3297743288 wait complete. 2024-04-29T20:23:04.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.156+0000 7ff1f40f2640 1 librados: shutdown 2024-04-29T20:23:04.169 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.4 2024-04-29T20:23:04.169 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:04.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.169+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb9a8018020 con 0x7fb9b80b2230 2024-04-29T20:23:04.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.170+0000 7fb9aeffd640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb99c06e2b0 0x7fb99c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.170+0000 7fb9be2c0640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb99c06e2b0 0x7fb99c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.170+0000 7fb9be2c0640 10 monclient: get_auth_request con 0x7fb99c06e2b0 auth_method 0 2024-04-29T20:23:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.170+0000 7fb9aeffd640 10 client.4387.objecter ms_handle_connect 0x7fb99c06e2b0 2024-04-29T20:23:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.170+0000 7fb9be2c0640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb99c06e2b0 0x7fb99c070770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40011e0 tx=0x7fb9b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:04.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.174+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7fa20c07a260 con 0x7fa21c0aa460 2024-04-29T20:23:04.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.174+0000 7fa218ff9640 10 client.4390.objecter ms_dispatch 0x7fa21c000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:04.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.174+0000 7fa218ff9640 3 client.4390.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:04.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.174+0000 7fa218ff9640 3 client.4390.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:04.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.174+0000 7fa218ff9640 20 client.4390.objecter dump_active .. 0 homeless 2024-04-29T20:23:04.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.179+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7fb9a8057e40 con 0x7fb9b80b2230 2024-04-29T20:23:04.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.179+0000 7fb9aeffd640 10 client.4387.objecter ms_dispatch 0x7fb9b8000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:04.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.179+0000 7fb9aeffd640 3 client.4387.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:04.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.179+0000 7fb9aeffd640 3 client.4387.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:04.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.179+0000 7fb9aeffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-04-29T20:23:04.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.199+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa20c07c8c0 con 0x7fa21c0aa460 2024-04-29T20:23:04.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.199+0000 7fa218ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:04.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.199+0000 7fa218ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:04.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.200+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9a801f020 con 0x7fb9b80b2230 2024-04-29T20:23:04.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.200+0000 7fb9aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:04.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.200+0000 7fb9aeffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:04.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.208+0000 7f867d7fa640 1 -- 172.21.15.64:0/530002946 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8678059330 con 0x7f86801cee90 2024-04-29T20:23:04.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.208+0000 7f867d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-29T20:23:04.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.208+0000 7f867d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:04.209 INFO:teuthology.orchestra.run.smithi064.stdout:94489280521 2024-04-29T20:23:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 10 librados: watch_flush enter 2024-04-29T20:23:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 10 librados: watch_flush exit 2024-04-29T20:23:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 20 client.4384.objecter shutdown clearing up homeless session... 2024-04-29T20:23:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 10 client.4384.objecter successfully canceled tick 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f866406e300 msgr2=0x7f86640707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f866406e300 0x7f86640707c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8674034000 tx=0x7f867402f000 comp rx=0 tx=0).stop 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 10 monclient: shutdown 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 msgr2=0x7f86801cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 0x7f86801cc770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f86800b5050 tx=0x7f867800c980 comp rx=0 tx=0).stop 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 shutdown_connections 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f866406e300 0x7f86640707c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86801cee90 0x7f86801cc770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86801cdcb0 0x7f86801ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 --2- 172.21.15.64:0/530002946 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f86700039b0 0x7f86801433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 >> 172.21.15.64:0/530002946 conn(0x7f868006c960 msgr2=0x7f86800b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 shutdown_connections 2024-04-29T20:23:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 -- 172.21.15.64:0/530002946 wait complete. 2024-04-29T20:23:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.210+0000 7f86871e6640 1 librados: shutdown 2024-04-29T20:23:04.224 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280521 got 94489280521 for osd.0 2024-04-29T20:23:04.224 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.397+0000 7fa222311640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-29T20:23:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.397+0000 7fa222311640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-29T20:23:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.397+0000 7fa222311640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.397+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa21c071f90 con 0x7fa21c0aa460 2024-04-29T20:23:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.402+0000 7fb9c0524640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-29T20:23:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.402+0000 7fb9c0524640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-29T20:23:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.402+0000 7fb9c0524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.402+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb9b81dc100 con 0x7fb9b80b2230 2024-04-29T20:23:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.443+0000 7fa218ff9640 1 -- 172.21.15.64:0/3065622774 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa20c07c1c0 con 0x7fa21c0aa460 2024-04-29T20:23:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.443+0000 7fa218ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-29T20:23:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.443+0000 7fa218ff9640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:04.445 INFO:teuthology.orchestra.run.smithi064.stdout:94489280522 2024-04-29T20:23:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.443+0000 7fb9aeffd640 1 -- 172.21.15.64:0/2993955931 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9a800f2e0 con 0x7fb9b80b2230 2024-04-29T20:23:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.443+0000 7fb9aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-29T20:23:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.443+0000 7fb9aeffd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:04.445 INFO:teuthology.orchestra.run.smithi064.stdout:94489280522 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 10 librados: watch_flush enter 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 10 librados: watch_flush exit 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 20 client.4390.objecter shutdown clearing up homeless session... 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 10 client.4390.objecter successfully canceled tick 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa20806e280 msgr2=0x7fa208070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa20806e280 0x7fa208070740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa210000f60 tx=0x7fa2100079d0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 10 monclient: shutdown 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 msgr2=0x7fa21c0ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 0x7fa21c0ad7c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009890 tx=0x7fa20c0046c0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 shutdown_connections 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa20806e280 0x7fa208070740 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa21c0b2230 0x7fa21c13ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa21c0b5c60 0x7fa21c13b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 --2- 172.21.15.64:0/3065622774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa21c0aa460 0x7fa21c0ad7c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 >> 172.21.15.64:0/3065622774 conn(0x7fa21c06c960 msgr2=0x7fa21c06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 10 librados: watch_flush enter 2024-04-29T20:23:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 10 librados: watch_flush exit 2024-04-29T20:23:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 20 client.4387.objecter shutdown clearing up homeless session... 2024-04-29T20:23:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 10 client.4387.objecter successfully canceled tick 2024-04-29T20:23:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 shutdown_connections 2024-04-29T20:23:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 -- 172.21.15.64:0/3065622774 wait complete. 2024-04-29T20:23:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb99c06e2b0 msgr2=0x7fb99c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb99c06e2b0 0x7fb99c070770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40011e0 tx=0x7fb9b4009040 comp rx=0 tx=0).stop 2024-04-29T20:23:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 10 monclient: shutdown 2024-04-29T20:23:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 msgr2=0x7fb9b81dc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 0x7fb9b81dc410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800b450 tx=0x7fb9a800b910 comp rx=0 tx=0).stop 2024-04-29T20:23:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fa222311640 1 librados: shutdown 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 shutdown_connections 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb99c06e2b0 0x7fb99c070770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb9b80b5c60 0x7fb9b8139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9b80b2230 0x7fb9b81dc410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 --2- 172.21.15.64:0/2993955931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb9b80aa460 0x7fb9b8143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 >> 172.21.15.64:0/2993955931 conn(0x7fb9b806c960 msgr2=0x7fb9b80b7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 shutdown_connections 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.445+0000 7fb9c0524640 1 -- 172.21.15.64:0/2993955931 wait complete. 2024-04-29T20:23:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.446+0000 7fb9c0524640 1 librados: shutdown 2024-04-29T20:23:04.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280522 got 94489280522 for osd.3 2024-04-29T20:23:04.460 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:04.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280522 got 94489280522 for osd.1 2024-04-29T20:23:04.460 DEBUG:teuthology.parallel:result is None 2024-04-29T20:23:04.461 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-29T20:23:04.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.558+0000 7f657cfd5640 10 monclient: get_monmap_and_config 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.558+0000 7f657cfd5640 10 monclient: build_initial_monmap 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.558+0000 7f657cfd5640 10 monclient: monmap: 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:04.559621+0000 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:04.559621+0000 2024-04-29T20:23:04.560 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:04.561 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:04.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.559+0000 7f657cfd5640 1 Processor -- start 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.559+0000 7f657cfd5640 1 -- start start 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.559+0000 7f657cfd5640 10 monclient: init 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65780b5f70 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6578143520 0x7f65780aa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient(hunting): picked mon.noname-c con 0x7f6578143520 addr v2:172.21.15.167:3300/0 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65780b1f00 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65780ac4b0 con 0x7f65780b5f70 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65780b2b70 con 0x7f65780b1f00 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f65780b6350 con 0x7f6578143520 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:04.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657cfd5640 10 monclient(hunting): authenticate will time out at 1533.719648s 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f656dd9b640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f6578143520 0x7f65780aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f65780b5f70 0x7f6578143ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41564/0 (socket says 172.21.15.64:41564) 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.560+0000 7f657659c640 1 -- 172.21.15.64:0/3400088532 learned_addr learned my addr 172.21.15.64:0/3400088532 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): get_auth_request con 0x7f6578143520 auth_method 0 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:04.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.561+0000 7f656dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f6575d9b640 1 -- 172.21.15.64:0/3400088532 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037666127 0 0) 0x7f65780ac4b0 con 0x7f65780b5f70 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f6575d9b640 10 monclient(hunting): _negotiate 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f6575d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f6575d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f6575d9b640 10 monclient(hunting): my global_id is 4395 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f6575d9b640 1 -- 172.21.15.64:0/3400088532 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6564003520 con 0x7f65780b5f70 2024-04-29T20:23:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 1 -- 172.21.15.64:0/3400088532 >> v1:172.21.15.167:6789/0 conn(0x7f65780b1f00 legacy=0x7f657814de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 1 -- 172.21.15.64:0/3400088532 >> v1:172.21.15.64:6789/0 conn(0x7f65780b5f70 legacy=0x7f6578143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 10 monclient: _finish_hunting 0 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 1 monclient: found mon.noname-c 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.563+0000 7f656dd9b640 1 -- 172.21.15.64:0/3400088532 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65780aa9a0 con 0x7f6578143520 2024-04-29T20:23:04.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.566+0000 7f656dd9b640 1 --2- 172.21.15.64:0/3400088532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6578143520 0x7f65780aa460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65600097a0 tx=0x7f656002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9173a8863ae9699 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:04.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 1 -- 172.21.15.64:0/3400088532 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65600315a0 con 0x7f6578143520 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: dump: 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:04.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: _finish_auth 0 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: _check_auth_tickets 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 1 -- 172.21.15.64:0/3400088532 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6560037070 con 0x7f6578143520 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 1 -- 172.21.15.64:0/3400088532 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560031960 con 0x7f6578143520 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f6575d9b640 10 monclient: dump: 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f657cfd5640 5 monclient: authenticate success, global_id 4393 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f657cfd5640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.570+0000 7f657cfd5640 4 set_mon_vals no callback set 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 10 monclient: shutdown 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 -- 172.21.15.64:0/3400088532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6578143520 msgr2=0x7f65780aa460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3400088532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6578143520 0x7f65780aa460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65600097a0 tx=0x7f656002b840 comp rx=0 tx=0).stop 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 -- 172.21.15.64:0/3400088532 shutdown_connections 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3400088532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6578143520 0x7f65780aa460 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 -- 172.21.15.64:0/3400088532 >> 172.21.15.64:0/3400088532 conn(0x7f657806cc10 msgr2=0x7f657806d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:04.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 -- 172.21.15.64:0/3400088532 shutdown_connections 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 1 -- 172.21.15.64:0/3400088532 wait complete. 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 10 monclient: build_initial_monmap 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.571+0000 7f657cfd5640 10 monclient: monmap: 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:04.578 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 1 librados: starting msgr at 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 1 librados: starting objecter 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 1 Processor -- start 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 1 -- start start 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 1 librados: setting wanted keys 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 1 librados: calling monclient init 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.572+0000 7f657cfd5640 10 monclient: init 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 0x7f6578146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): picked mon.noname-b con 0x7f65780b1f00 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65780b5f70 0x7f65780aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): picked mon.noname-a con 0x7f65780b5f70 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f65780b0bd0 0x7f65780b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): picked mon.noname-c con 0x7f65780b0bd0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6578140f10 con 0x7f65780b5f70 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6578143a40 con 0x7f65780b1f00 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f65780b2360 con 0x7f65780b0bd0 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:04.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657cfd5640 10 monclient(hunting): authenticate will time out at 1533.732680s 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f657659c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 0x7f6578146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f6576d9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f65780b0bd0 0x7f65780b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.573+0000 7f656dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65780b5f70 0x7f65780aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 0x7f6578146d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:54990/0 (socket says 172.21.15.64:54990) 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 1 -- 172.21.15.64:0/3631558613 learned_addr learned my addr 172.21.15.64:0/3631558613 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): get_auth_request con 0x7f65780b1f00 auth_method 0 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): get_auth_request con 0x7f65780b0bd0 auth_method 0 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): get_auth_request con 0x7f65780b5f70 auth_method 0 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f657659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f6576d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:04.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.574+0000 7f656dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:04.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-04-29T20:23:04.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 1 -- 172.21.15.64:0/3631558613 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f65780b0bd0 msgr2=0x7f65780b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f65780b0bd0 0x7f65780b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 1 -- 172.21.15.64:0/3631558613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65780b5f70 msgr2=0x7f65780aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65780b5f70 0x7f65780aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 10 monclient: _finish_hunting 0 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 1 monclient: found mon.noname-b 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.577+0000 7f657659c640 1 -- 172.21.15.64:0/3631558613 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65780b0540 con 0x7f65780b1f00 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.579+0000 7f657659c640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 0x7f6578146d20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f656800c460 tx=0x7f656800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.579+0000 7f656f7fe640 10 client.?.objecter ms_handle_connect 0x7f65780b1f00 2024-04-29T20:23:04.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.579+0000 7f656f7fe640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:04.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568017070 con 0x7f65780b1f00 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: dump: 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:04.586 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: _finish_auth 0 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 5 monclient: authenticate success, global_id 4396 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6568005ce0 con 0x7f65780b1f00 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 10 monclient: _renew_subs 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568007290 con 0x7f65780b1f00 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65780af180 con 0x7f65780b1f00 2024-04-29T20:23:04.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657559a640 4 set_mon_vals no callback set 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f656f7fe640 10 monclient: dump: 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:04.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 10 monclient: _renew_subs 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65780b15a0 con 0x7f65780b1f00 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.582+0000 7f657cfd5640 1 librados: init done 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.584+0000 7f657cfd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.584+0000 7f657cfd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.584+0000 7f657cfd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.584+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f657804fce0 con 0x7f65780b1f00 2024-04-29T20:23:04.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.602+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f656801a070 con 0x7f65780b1f00 2024-04-29T20:23:04.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.603+0000 7f656f7fe640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f655806e2d0 0x7f6558070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:04.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.603+0000 7f656dd9b640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f655806e2d0 0x7f6558070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:04.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.603+0000 7f656dd9b640 10 monclient: get_auth_request con 0x7f655806e2d0 auth_method 0 2024-04-29T20:23:04.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.603+0000 7f656f7fe640 10 client.4396.objecter ms_handle_connect 0x7f655806e2d0 2024-04-29T20:23:04.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.603+0000 7f656dd9b640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f655806e2d0 0x7f6558070790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6560036000 tx=0x7f656003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:04.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.605+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f6568057180 con 0x7f65780b1f00 2024-04-29T20:23:04.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.605+0000 7f656f7fe640 10 client.4396.objecter ms_dispatch 0x7f6578000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:04.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.605+0000 7f656f7fe640 3 client.4396.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:04.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.605+0000 7f656f7fe640 3 client.4396.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:04.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.605+0000 7f656f7fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-04-29T20:23:04.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.849+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6568021020 con 0x7f65780b1f00 2024-04-29T20:23:04.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.849+0000 7f656f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:04.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:04.849+0000 7f656f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:05.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.047+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f65780af350 con 0x7f655806e2d0 2024-04-29T20:23:05.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.049+0000 7f656f7fe640 1 -- 172.21.15.64:0/3631558613 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35520 (secure 0 0 0) 0x7f65780af350 con 0x7f655806e2d0 2024-04-29T20:23:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:23:05.051 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:23:05.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 10 librados: watch_flush enter 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 10 librados: watch_flush exit 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 20 client.4396.objecter shutdown clearing up homeless session... 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 10 client.4396.objecter successfully canceled tick 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f655806e2d0 msgr2=0x7f6558070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f655806e2d0 0x7f6558070790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6560036000 tx=0x7f656003d040 comp rx=0 tx=0).stop 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 10 monclient: shutdown 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 msgr2=0x7f6578146d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 0x7f6578146d20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f656800c460 tx=0x7f656800c920 comp rx=0 tx=0).stop 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 shutdown_connections 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f655806e2d0 0x7f6558070790 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f65780b0bd0 0x7f65780b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65780b5f70 0x7f65780aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 --2- 172.21.15.64:0/3631558613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65780b1f00 0x7f6578146d20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.051+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 >> 172.21.15.64:0/3631558613 conn(0x7f657806cc10 msgr2=0x7f65780b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.052+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 shutdown_connections 2024-04-29T20:23:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.052+0000 7f657cfd5640 1 -- 172.21.15.64:0/3631558613 wait complete. 2024-04-29T20:23:05.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.052+0000 7f657cfd5640 1 librados: shutdown 2024-04-29T20:23:05.064 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-29T20:23:03.186562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218884,"kb_used_data":2708,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755836,"statfs":{"total":773094113280,"available":772869976064,"internally_reserved":0,"allocated":2772992,"data_stored":1511774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.562872+0000","last_change":"2024-04-29T20:22:59.008821+0000","last_active":"2024-04-29T20:23:00.562872+0000","last_peered":"2024-04-29T20:23:00.562872+0000","last_clean":"2024-04-29T20:23:00.562872+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:23:00.562872+0000","last_undegraded":"2024-04-29T20:23:00.562872+0000","last_fullsized":"2024-04-29T20:23:00.562872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:05:01.384817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.124521+0000","last_change":"2024-04-29T20:22:59.053068+0000","last_active":"2024-04-29T20:23:00.124521+0000","last_peered":"2024-04-29T20:23:00.124521+0000","last_clean":"2024-04-29T20:23:00.124521+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:23:00.124521+0000","last_undegraded":"2024-04-29T20:23:00.124521+0000","last_fullsized":"2024-04-29T20:23:00.124521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:40:41.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.539389+0000","last_change":"2024-04-29T20:22:58.225857+0000","last_active":"2024-04-29T20:23:00.539389+0000","last_peered":"2024-04-29T20:23:00.539389+0000","last_clean":"2024-04-29T20:23:00.539389+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:23:00.539389+0000","last_undegraded":"2024-04-29T20:23:00.539389+0000","last_fullsized":"2024-04-29T20:23:00.539389+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:16:31.016297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.114296+0000","last_change":"2024-04-29T20:22:58.466585+0000","last_active":"2024-04-29T20:23:00.114296+0000","last_peered":"2024-04-29T20:23:00.114296+0000","last_clean":"2024-04-29T20:23:00.114296+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:23:00.114296+0000","last_undegraded":"2024-04-29T20:23:00.114296+0000","last_fullsized":"2024-04-29T20:23:00.114296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:05:28.648611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.467734+0000","last_change":"2024-04-29T20:22:58.456344+0000","last_active":"2024-04-29T20:23:00.467734+0000","last_peered":"2024-04-29T20:23:00.467734+0000","last_clean":"2024-04-29T20:23:00.467734+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:23:00.467734+0000","last_undegraded":"2024-04-29T20:23:00.467734+0000","last_fullsized":"2024-04-29T20:23:00.467734+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:02:31.378764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.708638+0000","last_change":"2024-04-29T20:22:58.100523+0000","last_active":"2024-04-29T20:23:00.708638+0000","last_peered":"2024-04-29T20:23:00.708638+0000","last_clean":"2024-04-29T20:23:00.708638+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:23:00.708638+0000","last_undegraded":"2024-04-29T20:23:00.708638+0000","last_fullsized":"2024-04-29T20:23:00.708638+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:31:35.619213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.453721+0000","last_change":"2024-04-29T20:22:58.155879+0000","last_active":"2024-04-29T20:23:00.453721+0000","last_peered":"2024-04-29T20:23:00.453721+0000","last_clean":"2024-04-29T20:23:00.453721+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:23:00.453721+0000","last_undegraded":"2024-04-29T20:23:00.453721+0000","last_fullsized":"2024-04-29T20:23:00.453721+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:40:44.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.534590+0000","last_change":"2024-04-29T20:22:58.874338+0000","last_active":"2024-04-29T20:23:00.534590+0000","last_peered":"2024-04-29T20:23:00.534590+0000","last_clean":"2024-04-29T20:23:00.534590+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:23:00.534590+0000","last_undegraded":"2024-04-29T20:23:00.534590+0000","last_fullsized":"2024-04-29T20:23:00.534590+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:59:51.304097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-29T20:22:59.260050+0000","last_change":"2024-04-29T20:22:57.765613+0000","last_active":"2024-04-29T20:22:59.260050+0000","last_peered":"2024-04-29T20:22:59.260050+0000","last_clean":"2024-04-29T20:22:59.260050+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:22:59.260050+0000","last_undegraded":"2024-04-29T20:22:59.260050+0000","last_fullsized":"2024-04-29T20:22:59.260050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_clean_scrub_stamp":"2024-04-29T20:22:42.263517+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:06:48.686106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":485154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280521,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":81904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":20,"seq":85899345929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":87314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":490902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":87314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:23:05.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.163+0000 7f5a0f915640 10 monclient: get_monmap_and_config 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.163+0000 7f5a0f915640 10 monclient: build_initial_monmap 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.164+0000 7f5a0f915640 10 monclient: monmap: 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:05.164853+0000 2024-04-29T20:23:05.165 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:05.164853+0000 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:05.166 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.164+0000 7f5a0f915640 1 Processor -- start 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.164+0000 7f5a0f915640 1 -- start start 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.164+0000 7f5a0f915640 10 monclient: init 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f5a080aeae0 0x7f5a08140670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): picked mon.noname-a con 0x7f5a080aeae0 addr v2:172.21.15.64:3300/0 2024-04-29T20:23:05.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a0806eeb0 0x7f5a08140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): picked mon.noname-b con 0x7f5a0806eeb0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a08143450 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a08143fb0 con 0x7f5a08143450 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f5a080aeec0 con 0x7f5a080aeae0 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0806f980 con 0x7f5a0806eeb0 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.165+0000 7f5a0f915640 10 monclient(hunting): authenticate will time out at 1534.324957s 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0d6b1640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f5a080aeae0 0x7f5a08140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0ceb0640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a0806eeb0 0x7f5a08140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0deb2640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f5a08143450 0x7f5a081438a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:43542/0 (socket says 172.21.15.64:43542) 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0deb2640 1 -- 172.21.15.64:0/2167977151 learned_addr learned my addr 172.21.15.64:0/2167977151 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0d6b1640 10 monclient(hunting): get_auth_request con 0x7f5a080aeae0 auth_method 0 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0d6b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0d6b1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0d6b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0ceb0640 10 monclient(hunting): get_auth_request con 0x7f5a0806eeb0 auth_method 0 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0ceb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0ceb0640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.166+0000 7f5a0ceb0640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.167+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.167+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.167+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.167+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.167+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.167+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a07fff640 1 -- 172.21.15.64:0/2167977151 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2323635359 0 0) 0x7f5a08143fb0 con 0x7f5a08143450 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a07fff640 10 monclient(hunting): _negotiate 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a07fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a07fff640 10 monclient(hunting): my global_id is 4399 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a07fff640 1 -- 172.21.15.64:0/2167977151 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f0003520 con 0x7f5a08143450 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 1 -- 172.21.15.64:0/2167977151 >> v2:172.21.15.64:3300/0 conn(0x7f5a080aeae0 msgr2=0x7f5a08140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 1 --2- 172.21.15.64:0/2167977151 >> v2:172.21.15.64:3300/0 conn(0x7f5a080aeae0 0x7f5a08140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 1 -- 172.21.15.64:0/2167977151 >> v1:172.21.15.167:6789/0 conn(0x7f5a08143450 legacy=0x7f5a081438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 10 monclient: _finish_hunting 0 2024-04-29T20:23:05.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 1 monclient: found mon.noname-b 2024-04-29T20:23:05.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:05.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:23:05.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.168+0000 7f5a0ceb0640 1 -- 172.21.15.64:0/2167977151 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a080706f0 con 0x7f5a0806eeb0 2024-04-29T20:23:05.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.170+0000 7f5a0ceb0640 1 --2- 172.21.15.64:0/2167977151 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a0806eeb0 0x7f5a08140bb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0097a0 tx=0x7f59fc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3169cfc26de74bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 1 -- 172.21.15.64:0/2167977151 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0315a0 con 0x7f5a0806eeb0 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: dump: 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: _finish_auth 0 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: _check_auth_tickets 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a0f915640 5 monclient: authenticate success, global_id 4289 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 1 -- 172.21.15.64:0/2167977151 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc037070 con 0x7f5a0806eeb0 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a0f915640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a0f915640 4 set_mon_vals no callback set 2024-04-29T20:23:05.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 1 -- 172.21.15.64:0/2167977151 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc031960 con 0x7f5a0806eeb0 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.174+0000 7f5a07fff640 10 monclient: dump: 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.178 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 10 monclient: shutdown 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 -- 172.21.15.64:0/2167977151 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a0806eeb0 msgr2=0x7f5a08140bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 --2- 172.21.15.64:0/2167977151 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a0806eeb0 0x7f5a08140bb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0097a0 tx=0x7f59fc02b840 comp rx=0 tx=0).stop 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 -- 172.21.15.64:0/2167977151 shutdown_connections 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 --2- 172.21.15.64:0/2167977151 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a0806eeb0 0x7f5a08140bb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 --2- 172.21.15.64:0/2167977151 >> v2:172.21.15.64:3300/0 conn(0x7f5a080aeae0 0x7f5a08140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 -- 172.21.15.64:0/2167977151 >> 172.21.15.64:0/2167977151 conn(0x7f5a080b50a0 msgr2=0x7f5a080b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 -- 172.21.15.64:0/2167977151 shutdown_connections 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.175+0000 7f5a0f915640 1 -- 172.21.15.64:0/2167977151 wait complete. 2024-04-29T20:23:05.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 10 monclient: build_initial_monmap 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 10 monclient: monmap: 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 1 librados: starting msgr at 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 1 librados: starting objecter 2024-04-29T20:23:05.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 1 Processor -- start 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 1 -- start start 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 1 librados: setting wanted keys 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 1 librados: calling monclient init 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.176+0000 7f5a0f915640 10 monclient: init 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a0806eeb0 0x7f5a08139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): picked mon.noname-b con 0x7f5a0806eeb0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a080aeae0 0x7f5a081dc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): picked mon.noname-a con 0x7f5a080aeae0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a08143450 0x7f5a081dc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): picked mon.noname-c con 0x7f5a08143450 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0806f460 con 0x7f5a080aeae0 2024-04-29T20:23:05.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5a08070cb0 con 0x7f5a0806eeb0 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0814dd60 con 0x7f5a08143450 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0f915640 10 monclient(hunting): authenticate will time out at 1534.337018s 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.177+0000 7f5a0deb2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a08143450 0x7f5a081dc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a0806eeb0 0x7f5a08139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a080aeae0 0x7f5a081dc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a08143450 0x7f5a081dc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:40564/0 (socket says 172.21.15.64:40564) 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 1 -- 172.21.15.64:0/293715605 learned_addr learned my addr 172.21.15.64:0/293715605 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): get_auth_request con 0x7f5a08143450 auth_method 0 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): get_auth_request con 0x7f5a080aeae0 auth_method 0 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): get_auth_request con 0x7f5a0806eeb0 auth_method 0 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0deb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.178+0000 7f5a0ceb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 1 -- 172.21.15.64:0/293715605 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a08143450 msgr2=0x7f5a081dc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a08143450 0x7f5a081dc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 1 -- 172.21.15.64:0/293715605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a080aeae0 msgr2=0x7f5a081dc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a080aeae0 0x7f5a081dc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 10 monclient: _finish_hunting 0 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 1 monclient: found mon.noname-b 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.181+0000 7f5a0d6b1640 1 -- 172.21.15.64:0/293715605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0813b5f0 con 0x7f5a0806eeb0 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.183+0000 7f5a0d6b1640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a0806eeb0 0x7f5a08139fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f59f800efc0 tx=0x7f59f800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.183+0000 7f5a05ffb640 10 client.?.objecter ms_handle_connect 0x7f5a0806eeb0 2024-04-29T20:23:05.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.183+0000 7f5a05ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8018020 con 0x7f5a0806eeb0 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: dump: 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: _finish_auth 0 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f8014480 con 0x7f5a0806eeb0 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 5 monclient: authenticate success, global_id 4402 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8014850 con 0x7f5a0806eeb0 2024-04-29T20:23:05.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a05ffb640 10 monclient: dump: 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 10 monclient: _renew_subs 2024-04-29T20:23:05.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0813bc80 con 0x7f5a0806eeb0 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a077fe640 4 set_mon_vals no callback set 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 10 monclient: _renew_subs 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0813a6c0 con 0x7f5a0806eeb0 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.187+0000 7f5a0f915640 1 librados: init done 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.188+0000 7f5a0f915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.188+0000 7f5a0f915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.188+0000 7f5a0f915640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:23:05.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.188+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0804fce0 con 0x7f5a0806eeb0 2024-04-29T20:23:05.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.206+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f59f8005d10 con 0x7f5a0806eeb0 2024-04-29T20:23:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.206+0000 7f5a05ffb640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f59f406e300 0x7f59f40707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.206+0000 7f5a0ceb0640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f59f406e300 0x7f59f40707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.206+0000 7f5a0ceb0640 10 monclient: get_auth_request con 0x7f59f406e300 auth_method 0 2024-04-29T20:23:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.207+0000 7f5a05ffb640 10 client.4402.objecter ms_handle_connect 0x7f59f406e300 2024-04-29T20:23:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.207+0000 7f5a0ceb0640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f59f406e300 0x7f59f40707c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f59fc036000 tx=0x7f59fc03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:05.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.209+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f59f8058c50 con 0x7f5a0806eeb0 2024-04-29T20:23:05.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.209+0000 7f5a05ffb640 10 client.4402.objecter ms_dispatch 0x7f5a08000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:05.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.209+0000 7f5a05ffb640 3 client.4402.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:05.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.209+0000 7f5a05ffb640 3 client.4402.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:05.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.209+0000 7f5a05ffb640 20 client.4402.objecter dump_active .. 0 homeless 2024-04-29T20:23:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.455+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59f8022030 con 0x7f5a0806eeb0 2024-04-29T20:23:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.455+0000 7f5a05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:05.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.455+0000 7f5a05ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:05.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.645+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a081dbd10 con 0x7f59f406e300 2024-04-29T20:23:05.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.646+0000 7f5a05ffb640 1 -- 172.21.15.64:0/293715605 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35524 (secure 0 0 0) 0x7f5a081dbd10 con 0x7f59f406e300 2024-04-29T20:23:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:23:05.648 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:23:05.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 10 librados: watch_flush enter 2024-04-29T20:23:05.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 10 librados: watch_flush exit 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 20 client.4402.objecter shutdown clearing up homeless session... 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 10 client.4402.objecter successfully canceled tick 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f59f406e300 msgr2=0x7f59f40707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f59f406e300 0x7f59f40707c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f59fc036000 tx=0x7f59fc03d040 comp rx=0 tx=0).stop 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 10 monclient: shutdown 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a0806eeb0 msgr2=0x7f5a08139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.648+0000 7f5a0f915640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a0806eeb0 0x7f5a08139fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f59f800efc0 tx=0x7f59f800c490 comp rx=0 tx=0).stop 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 shutdown_connections 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f59f406e300 0x7f59f40707c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5a08143450 0x7f5a081dc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a080aeae0 0x7f5a081dc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 --2- 172.21.15.64:0/293715605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a0806eeb0 0x7f5a08139fa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 >> 172.21.15.64:0/293715605 conn(0x7f5a080b50a0 msgr2=0x7f5a0813ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:05.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 shutdown_connections 2024-04-29T20:23:05.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 -- 172.21.15.64:0/293715605 wait complete. 2024-04-29T20:23:05.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.649+0000 7f5a0f915640 1 librados: shutdown 2024-04-29T20:23:05.661 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-29T20:23:05.186871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218940,"kb_used_data":2756,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755780,"statfs":{"total":773094113280,"available":772869918720,"internally_reserved":0,"allocated":2822144,"data_stored":1534428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.401594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.562872+0000","last_change":"2024-04-29T20:22:59.008821+0000","last_active":"2024-04-29T20:23:00.562872+0000","last_peered":"2024-04-29T20:23:00.562872+0000","last_clean":"2024-04-29T20:23:00.562872+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:23:00.562872+0000","last_undegraded":"2024-04-29T20:23:00.562872+0000","last_fullsized":"2024-04-29T20:23:00.562872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:05:01.384817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.124521+0000","last_change":"2024-04-29T20:22:59.053068+0000","last_active":"2024-04-29T20:23:00.124521+0000","last_peered":"2024-04-29T20:23:00.124521+0000","last_clean":"2024-04-29T20:23:00.124521+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:23:00.124521+0000","last_undegraded":"2024-04-29T20:23:00.124521+0000","last_fullsized":"2024-04-29T20:23:00.124521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:40:41.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.539389+0000","last_change":"2024-04-29T20:22:58.225857+0000","last_active":"2024-04-29T20:23:00.539389+0000","last_peered":"2024-04-29T20:23:00.539389+0000","last_clean":"2024-04-29T20:23:00.539389+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:23:00.539389+0000","last_undegraded":"2024-04-29T20:23:00.539389+0000","last_fullsized":"2024-04-29T20:23:00.539389+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:16:31.016297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.114296+0000","last_change":"2024-04-29T20:22:58.466585+0000","last_active":"2024-04-29T20:23:00.114296+0000","last_peered":"2024-04-29T20:23:00.114296+0000","last_clean":"2024-04-29T20:23:00.114296+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:23:00.114296+0000","last_undegraded":"2024-04-29T20:23:00.114296+0000","last_fullsized":"2024-04-29T20:23:00.114296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:05:28.648611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.467734+0000","last_change":"2024-04-29T20:22:58.456344+0000","last_active":"2024-04-29T20:23:00.467734+0000","last_peered":"2024-04-29T20:23:00.467734+0000","last_clean":"2024-04-29T20:23:00.467734+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:23:00.467734+0000","last_undegraded":"2024-04-29T20:23:00.467734+0000","last_fullsized":"2024-04-29T20:23:00.467734+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:02:31.378764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.708638+0000","last_change":"2024-04-29T20:22:58.100523+0000","last_active":"2024-04-29T20:23:00.708638+0000","last_peered":"2024-04-29T20:23:00.708638+0000","last_clean":"2024-04-29T20:23:00.708638+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:23:00.708638+0000","last_undegraded":"2024-04-29T20:23:00.708638+0000","last_fullsized":"2024-04-29T20:23:00.708638+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:31:35.619213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.453721+0000","last_change":"2024-04-29T20:22:58.155879+0000","last_active":"2024-04-29T20:23:00.453721+0000","last_peered":"2024-04-29T20:23:00.453721+0000","last_clean":"2024-04-29T20:23:00.453721+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:23:00.453721+0000","last_undegraded":"2024-04-29T20:23:00.453721+0000","last_fullsized":"2024-04-29T20:23:00.453721+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:40:44.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.534590+0000","last_change":"2024-04-29T20:22:58.874338+0000","last_active":"2024-04-29T20:23:00.534590+0000","last_peered":"2024-04-29T20:23:00.534590+0000","last_clean":"2024-04-29T20:23:00.534590+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:23:00.534590+0000","last_undegraded":"2024-04-29T20:23:00.534590+0000","last_fullsized":"2024-04-29T20:23:00.534590+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:59:51.304097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-29T20:23:00.728322+0000","last_change":"2024-04-29T20:22:57.765613+0000","last_active":"2024-04-29T20:23:00.728322+0000","last_peered":"2024-04-29T20:23:00.728322+0000","last_clean":"2024-04-29T20:23:00.728322+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:23:00.728322+0000","last_undegraded":"2024-04-29T20:23:00.728322+0000","last_fullsized":"2024-04-29T20:23:00.728322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_clean_scrub_stamp":"2024-04-29T20:22:42.263517+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:06:48.686106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280523,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":655360,"data_stored":490902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":20,"seq":85899345930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":490902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":93062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":87314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:23:05.661 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-29T20:23:05.662 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-29T20:23:05.662 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-29T20:23:05.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-29T20:23:05.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.760+0000 7f44a4f11640 10 monclient: get_monmap_and_config 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.760+0000 7f44a4f11640 10 monclient: build_initial_monmap 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.760+0000 7f44a4f11640 10 monclient: monmap: 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:23:05.761337+0000 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:23:05.761337+0000 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:23:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:23:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:23:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:23:05.763 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 1 Processor -- start 2024-04-29T20:23:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 1 -- start start 2024-04-29T20:23:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient: init 2024-04-29T20:23:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00b22a0 0x7f44a0146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient(hunting): picked mon.noname-b con 0x7f44a00b22a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44a00b2c20 addr v1:172.21.15.64:6789/0 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44a00aa6a0 addr v1:172.21.15.167:6789/0 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44a00ac1f0 con 0x7f44a00b2c20 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44a00ac4d0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f44a00b33d0 con 0x7f44a00b22a0 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.761+0000 7f44a4f11640 10 monclient(hunting): authenticate will time out at 1534.921440s 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f4497fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f44a00b2c20 0x7f44a00b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41566/0 (socket says 172.21.15.64:41566) 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f4497fff640 1 -- 172.21.15.64:0/1235276651 learned_addr learned my addr 172.21.15.64:0/1235276651 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f449e59c640 1 --2- 172.21.15.64:0/1235276651 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00b22a0 0x7f44a0146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f449e59c640 10 monclient(hunting): get_auth_request con 0x7f44a00b22a0 auth_method 0 2024-04-29T20:23:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f449e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f449e59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.762+0000 7f449e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.763+0000 7f449e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.763+0000 7f449e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.763+0000 7f449e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 1 -- 172.21.15.64:0/1235276651 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354246786 0 0) 0x7f44a00ac1f0 con 0x7f44a00b2c20 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 10 monclient(hunting): _negotiate 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 10 monclient(hunting): my global_id is 4407 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 1 -- 172.21.15.64:0/1235276651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f448c003520 con 0x7f44a00b2c20 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 1 -- 172.21.15.64:0/1235276651 >> v1:172.21.15.167:6789/0 conn(0x7f44a00aa6a0 legacy=0x7f44a014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 1 -- 172.21.15.64:0/1235276651 >> v1:172.21.15.64:6789/0 conn(0x7f44a00b2c20 legacy=0x7f44a00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 1 -- 172.21.15.64:0/1235276651 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040167331 0 0) 0x7f44a00ac4d0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 10 monclient: _finish_hunting 0 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 1 monclient: found mon.noname-b 2024-04-29T20:23:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:23:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449e59c640 1 -- 172.21.15.64:0/1235276651 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a00b3040 con 0x7f44a00b22a0 2024-04-29T20:23:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.765+0000 7f449dd9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-29T20:23:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.767+0000 7f449e59c640 1 --2- 172.21.15.64:0/1235276651 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00b22a0 0x7f44a0146a40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44a00b3550 tx=0x7f449002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9261a0c3457ee4b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 1 -- 172.21.15.64:0/1235276651 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490035020 con 0x7f44a00b22a0 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: dump: 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.772 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: _finish_auth 0 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: _check_auth_tickets 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f44a4f11640 5 monclient: authenticate success, global_id 4295 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f44a4f11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 1 -- 172.21.15.64:0/1235276651 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4490031890 con 0x7f44a00b22a0 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 1 -- 172.21.15.64:0/1235276651 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490031c40 con 0x7f44a00b22a0 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f449dd9b640 10 monclient: dump: 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f44a4f11640 10 monclient: get_monmap_and_config success 2024-04-29T20:23:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.770+0000 7f44a4f11640 4 set_mon_vals no callback set 2024-04-29T20:23:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 10 monclient: shutdown 2024-04-29T20:23:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 -- 172.21.15.64:0/1235276651 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00b22a0 msgr2=0x7f44a0146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 --2- 172.21.15.64:0/1235276651 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00b22a0 0x7f44a0146a40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44a00b3550 tx=0x7f449002b930 comp rx=0 tx=0).stop 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 -- 172.21.15.64:0/1235276651 shutdown_connections 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 --2- 172.21.15.64:0/1235276651 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00b22a0 0x7f44a0146a40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 -- 172.21.15.64:0/1235276651 >> 172.21.15.64:0/1235276651 conn(0x7f44a006c940 msgr2=0x7f44a006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 -- 172.21.15.64:0/1235276651 shutdown_connections 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.771+0000 7f44a4f11640 1 -- 172.21.15.64:0/1235276651 wait complete. 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 10 monclient: build_initial_monmap 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 10 monclient: monmap: 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 1 librados: starting msgr at 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 1 librados: starting objecter 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 1 Processor -- start 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 1 -- start start 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 1 librados: setting wanted keys 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 1 librados: calling monclient init 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.772+0000 7f44a4f11640 10 monclient: init 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient: _reopen_session rank -1 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 0x7f44a0137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): picked mon.noname-c con 0x7f44a00aa6a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:23:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44a00b22a0 0x7f44a0141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): picked mon.noname-a con 0x7f44a00b22a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:23:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a00b2c20 0x7f44a013de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): picked mon.noname-b con 0x7f44a00b2c20 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:23:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f44a00aacb0 con 0x7f44a00b22a0 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44a00aab30 con 0x7f44a00b2c20 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): start opening mon connection 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f44a014f4f0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): _renew_subs 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.773+0000 7f44a4f11640 10 monclient(hunting): authenticate will time out at 1534.932937s 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44a00b22a0 0x7f44a0141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 0x7f44a0137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449ed9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a00b2c20 0x7f44a013de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 0x7f44a0137470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:40570/0 (socket says 172.21.15.64:40570) 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 1 -- 172.21.15.64:0/2936997364 learned_addr learned my addr 172.21.15.64:0/2936997364 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): get_auth_request con 0x7f44a00aa6a0 auth_method 0 2024-04-29T20:23:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): get_auth_request con 0x7f44a00b22a0 auth_method 0 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449ed9d640 10 monclient(hunting): get_auth_request con 0x7f44a00b2c20 auth_method 0 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f449e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.774+0000 7f4497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.775+0000 7f449ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:23:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.775+0000 7f449ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:23:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.775+0000 7f449ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 1 -- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a00b2c20 msgr2=0x7f44a013de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a00b2c20 0x7f44a013de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 1 -- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44a00b22a0 msgr2=0x7f44a0141340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44a00b22a0 0x7f44a0141340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 10 monclient: _finish_hunting 0 2024-04-29T20:23:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 1 monclient: found mon.noname-c 2024-04-29T20:23:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:23:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:23:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.777+0000 7f449e59c640 1 -- 172.21.15.64:0/2936997364 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a013d9f0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.780+0000 7f449e59c640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 0x7f44a0137470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4490000c00 tx=0x7f44900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.780+0000 7f4496ffd640 10 client.?.objecter ms_handle_connect 0x7f44a00aa6a0 2024-04-29T20:23:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.780+0000 7f4496ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:23:05.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490035040 con 0x7f44a00aa6a0 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: dump: 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: _finish_auth 0 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 5 monclient: authenticate success, global_id 4298 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4490003820 con 0x7f44a00aa6a0 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 10 monclient: _renew_subs 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a013e350 con 0x7f44a00aa6a0 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490003c10 con 0x7f44a00aa6a0 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f4496ffd640 10 monclient: dump: 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:23:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f449d59a640 4 set_mon_vals no callback set 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 10 monclient: _renew_subs 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a0141fa0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.784+0000 7f44a4f11640 1 librados: init done 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.785+0000 7f44a4f11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.785+0000 7f44a4f11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:05.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.785+0000 7f44a4f11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:05.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.785+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a00b29e0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.803+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4490038020 con 0x7f44a00aa6a0 2024-04-29T20:23:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.803+0000 7f4496ffd640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f448006e2d0 0x7f4480070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:23:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.804+0000 7f4497fff640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f448006e2d0 0x7f4480070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:23:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.804+0000 7f4497fff640 10 monclient: get_auth_request con 0x7f448006e2d0 auth_method 0 2024-04-29T20:23:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.804+0000 7f4496ffd640 10 client.4298.objecter ms_handle_connect 0x7f448006e2d0 2024-04-29T20:23:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.804+0000 7f4497fff640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f448006e2d0 0x7f4480070790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f44a0137c20 tx=0x7f4488009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:23:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.806+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7f449007a2c0 con 0x7f44a00aa6a0 2024-04-29T20:23:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.806+0000 7f4496ffd640 10 client.4298.objecter ms_dispatch 0x7f44a0000b90 osd_map(27..27 src has 1..27) 2024-04-29T20:23:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.806+0000 7f4496ffd640 3 client.4298.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:23:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.806+0000 7f4496ffd640 3 client.4298.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:23:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:05.806+0000 7f4496ffd640 20 client.4298.objecter dump_active .. 0 homeless 2024-04-29T20:23:06.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.067+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f44a00b29e0 con 0x7f44a00aa6a0 2024-04-29T20:23:06.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.067+0000 7f4496ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:23:06.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.067+0000 7f4496ffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:23:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.294+0000 7f44a4f11640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-29T20:23:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.294+0000 7f44a4f11640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-29T20:23:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.294+0000 7f44a4f11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:23:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.294+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f44a004fce0 con 0x7f44a00aa6a0 2024-04-29T20:23:06.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.300+0000 7f4496ffd640 1 -- 172.21.15.64:0/2936997364 <== mon.2 v2:172.21.15.64:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f449002d040 con 0x7f44a00aa6a0 2024-04-29T20:23:06.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.300+0000 7f4496ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-29T20:23:06.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.300+0000 7f4496ffd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:23:06.302 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:23:06.302 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-29T20:23:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 10 librados: watch_flush enter 2024-04-29T20:23:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 10 librados: watch_flush exit 2024-04-29T20:23:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 20 client.4298.objecter shutdown clearing up homeless session... 2024-04-29T20:23:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 10 client.4298.objecter successfully canceled tick 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f448006e2d0 msgr2=0x7f4480070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f448006e2d0 0x7f4480070790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f44a0137c20 tx=0x7f4488009040 comp rx=0 tx=0).stop 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 10 monclient: shutdown 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 msgr2=0x7f44a0137470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.302+0000 7f44a4f11640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 0x7f44a0137470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4490000c00 tx=0x7f44900046c0 comp rx=0 tx=0).stop 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 shutdown_connections 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f448006e2d0 0x7f4480070790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a00b2c20 0x7f44a013de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44a00b22a0 0x7f44a0141340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 --2- 172.21.15.64:0/2936997364 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f44a00aa6a0 0x7f44a0137470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 >> 172.21.15.64:0/2936997364 conn(0x7f44a006c940 msgr2=0x7f44a0146570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 shutdown_connections 2024-04-29T20:23:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 -- 172.21.15.64:0/2936997364 wait complete. 2024-04-29T20:23:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:23:06.303+0000 7f44a4f11640 1 librados: shutdown 2024-04-29T20:23:06.315 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-29T20:23:06.316 INFO:teuthology.run_tasks:Running task workunit... 2024-04-29T20:23:06.329 INFO:tasks.workunit:Pulling workunits from ref 354447ca5357e926795d009be846687f04556beb 2024-04-29T20:23:06.330 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-29T20:23:06.330 DEBUG:teuthology.orchestra.run.smithi064:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-29T20:23:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:23:06.346 INFO:teuthology.orchestra.run.smithi064.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-29T20:23:06.347 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-29T20:23:06.402 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-29T20:23:06.402 DEBUG:teuthology.orchestra.run.smithi064:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-29T20:23:06.457 INFO:tasks.workunit:timeout=6h 2024-04-29T20:23:06.457 INFO:tasks.workunit:cleanup=True 2024-04-29T20:23:06.458 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 354447ca5357e926795d009be846687f04556beb 2024-04-29T20:23:06.513 INFO:tasks.workunit.client.0.smithi064.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-29T20:23:11.087 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:29.593 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.091 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:29.597 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.092 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:29.569 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.095 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:29.572 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.138 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:29.644 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.140 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:29.616 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.143 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:29.649 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.147 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:29.653 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:11.189 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:29.694 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.128 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:49.633 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.132 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:49.638 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.146 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:49.652 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.181 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:49.658 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.182 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:49.688 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.183 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:49.660 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.185 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:49.691 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.197 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:49.673 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:31.202 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:49.708 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.169 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:09.675 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.175 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:09.681 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.190 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:09.696 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.227 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:09.733 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.230 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:09.707 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.231 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:09.708 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.232 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:09.738 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.246 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:09.722 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:23:51.249 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:09.755 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.218 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:29.722 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.226 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:29.732 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.242 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:29.748 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.276 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:29.752 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.277 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:29.754 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.281 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:29.787 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.286 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:29.792 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.292 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:29.769 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:11.301 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:29.807 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.249 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:49.755 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.303 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:49.779 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.307 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:49.784 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.324 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:49.801 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.339 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:49.845 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.361 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:49.867 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.421 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:49.927 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.433 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:49.939 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:31.456 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:49.962 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:48.999 INFO:tasks.workunit.client.0.smithi064.stderr:Updating files: 92% (11361/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-29T20:24:49.284 INFO:tasks.workunit.client.0.smithi064.stderr:Note: switching to '354447ca5357e926795d009be846687f04556beb'. 2024-04-29T20:24:49.285 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.285 INFO:tasks.workunit.client.0.smithi064.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-29T20:24:49.285 INFO:tasks.workunit.client.0.smithi064.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-29T20:24:49.285 INFO:tasks.workunit.client.0.smithi064.stderr:state without impacting any branches by switching back to a branch. 2024-04-29T20:24:49.285 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.286 INFO:tasks.workunit.client.0.smithi064.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-29T20:24:49.286 INFO:tasks.workunit.client.0.smithi064.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-29T20:24:49.286 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.287 INFO:tasks.workunit.client.0.smithi064.stderr: git switch -c 2024-04-29T20:24:49.287 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.287 INFO:tasks.workunit.client.0.smithi064.stderr:Or undo this operation with: 2024-04-29T20:24:49.287 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.287 INFO:tasks.workunit.client.0.smithi064.stderr: git switch - 2024-04-29T20:24:49.287 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.288 INFO:tasks.workunit.client.0.smithi064.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-29T20:24:49.288 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-29T20:24:49.288 INFO:tasks.workunit.client.0.smithi064.stderr:HEAD is now at 354447ca535 Merge branch 'wip-nitzan-public-addr-check-subnet-unreachable' of https://github.com/NitzanMordhai/ceph into wip-yuri6-testing-2024-04-17-1820 2024-04-29T20:24:49.293 DEBUG:teuthology.orchestra.run.smithi064:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-29T20:24:49.356 INFO:tasks.workunit.client.0.smithi064.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-29T20:24:49.358 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-29T20:24:49.359 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-29T20:24:49.846 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-29T20:24:49.900 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-29T20:24:49.941 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-29T20:24:49.942 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-29T20:24:49.943 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-29T20:24:49.982 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-29T20:24:49.985 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-29T20:24:49.985 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-29T20:24:50.041 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-29T20:24:50.042 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-29T20:24:50.042 DEBUG:teuthology.orchestra.run.smithi064:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=354447ca5357e926795d009be846687f04556beb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-29T20:24:50.100 INFO:tasks.workunit.client.0.smithi064.stderr:+ parallel=1 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' '' = --serial ']' 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ crimson=0 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' '' = --crimson ']' 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ color= 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -t 1 ']' 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-29T20:24:50.101 INFO:tasks.workunit.client.0.smithi064.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-29T20:24:50.102 INFO:tasks.workunit.client.0.smithi064.stderr:+ declare -A pids 2024-04-29T20:24:50.102 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.102 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.102 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_aio 2024-04-29T20:24:50.102 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_aio' 2024-04-29T20:24:50.103 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_aio 2024-04-29T20:24:50.103 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.104 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_aio 2024-04-29T20:24:50.104 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38097 2024-04-29T20:24:50.104 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_aio on pid 38097' 2024-04-29T20:24:50.104 INFO:tasks.workunit.client.0.smithi064.stdout:test api_aio on pid 38097 2024-04-29T20:24:50.107 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38097 2024-04-29T20:24:50.107 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_aio_pp 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_aio_pp' 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_aio_pp 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.108 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.109 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_aio_pp 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38104 2024-04-29T20:24:50.110 INFO:tasks.workunit.client.0.smithi064.stdout:test api_aio_pp on pid 38104 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_aio_pp on pid 38104' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38104 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_io 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_io' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_io 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.115 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.116 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.117 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_io 2024-04-29T20:24:50.118 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38113 2024-04-29T20:24:50.119 INFO:tasks.workunit.client.0.smithi064.stdout:test api_io on pid 38113 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_io on pid 38113' 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38113 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_io_pp 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_io_pp' 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_io_pp 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.124 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.125 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.126 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.127 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_io_pp 2024-04-29T20:24:50.128 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38124 2024-04-29T20:24:50.129 INFO:tasks.workunit.client.0.smithi064.stdout:test api_io_pp on pid 38124 2024-04-29T20:24:50.133 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_io_pp on pid 38124' 2024-04-29T20:24:50.133 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38124 2024-04-29T20:24:50.133 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.133 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.133 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-29T20:24:50.133 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_asio 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_asio' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_asio 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.134 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.135 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.136 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.137 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.138 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_asio 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38137 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_asio on pid 38137' 2024-04-29T20:24:50.139 INFO:tasks.workunit.client.0.smithi064.stdout:test api_asio on pid 38137 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38137 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_list 2024-04-29T20:24:50.147 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_list' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_list 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.148 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.149 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.150 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.151 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.153 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.154 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_list 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.155 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38153 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_list on pid 38153' 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.156 INFO:tasks.workunit.client.0.smithi064.stdout:test api_list on pid 38153 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38153 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.162 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_lock 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_lock' 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38097/exe 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_lock 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.163 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.164 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.165 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.166 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.167 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.168 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.169 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.170 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_aio: /' 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38104/exe 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_lock 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38175 2024-04-29T20:24:50.171 INFO:tasks.workunit.client.0.smithi064.stdout:test api_lock on pid 38175 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_lock on pid 38175' 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38175 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.179 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_lock_pp 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_lock_pp' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.180 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.181 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.182 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.183 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_lock_pp 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.184 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.185 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.186 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38113/exe 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.187 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_lock_pp 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38197 2024-04-29T20:24:50.188 INFO:tasks.workunit.client.0.smithi064.stdout:test api_lock_pp on pid 38197 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_lock_pp on pid 38197' 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38197 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.196 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.197 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_misc 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.198 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_misc' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_io.log 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_misc 2024-04-29T20:24:50.199 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_io: /' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.200 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.201 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.202 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38124/exe 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_misc 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.203 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38216 2024-04-29T20:24:50.204 INFO:tasks.workunit.client.0.smithi064.stdout:test api_misc on pid 38216 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_misc on pid 38216' 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38216 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.216 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.217 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.218 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_misc_pp 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_misc_pp' 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.219 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.220 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.221 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38137/exe 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.222 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.223 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_misc_pp 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.224 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.225 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_asio: /' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.226 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_misc_pp 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38241 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_misc_pp on pid 38241' 2024-04-29T20:24:50.227 INFO:tasks.workunit.client.0.smithi064.stdout:test api_misc_pp on pid 38241 2024-04-29T20:24:50.244 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38241 2024-04-29T20:24:50.244 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.245 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.246 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_tier_pp 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.247 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_tier_pp' 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.248 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38153/exe 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.249 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.250 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_tier_pp 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.251 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.252 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_list: /' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.253 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_list.log 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38175/exe 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.254 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.255 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.256 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_lock: /' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.257 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.258 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_tier_pp 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38197/exe 2024-04-29T20:24:50.259 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38277 2024-04-29T20:24:50.260 INFO:tasks.workunit.client.0.smithi064.stdout:test api_tier_pp on pid 38277 2024-04-29T20:24:50.269 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_tier_pp on pid 38277' 2024-04-29T20:24:50.269 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38277 2024-04-29T20:24:50.269 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.269 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.269 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.270 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.271 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_pool 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_pool' 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.272 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.273 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.274 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38216/exe 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.275 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_pool 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.276 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.277 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_pool 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38334 2024-04-29T20:24:50.278 INFO:tasks.workunit.client.0.smithi064.stdout:test api_pool on pid 38334 2024-04-29T20:24:50.282 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_pool on pid 38334' 2024-04-29T20:24:50.282 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38334 2024-04-29T20:24:50.282 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_snapshots 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_snapshots' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.283 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.284 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_snapshots 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38241/exe 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.285 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_misc: /' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_snapshots 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38354 2024-04-29T20:24:50.286 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_snapshots on pid 38354' 2024-04-29T20:24:50.287 INFO:tasks.workunit.client.0.smithi064.stdout:test api_snapshots on pid 38354 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38354 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.291 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.292 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.293 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_snapshots_pp 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_snapshots_pp' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.294 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.295 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.295 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_snapshots_pp 2024-04-29T20:24:50.295 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-29T20:24:50.295 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_snapshots_pp 2024-04-29T20:24:50.295 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38366 2024-04-29T20:24:50.295 INFO:tasks.workunit.client.0.smithi064.stdout:test api_snapshots_pp on pid 38366 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_snapshots_pp on pid 38366' 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38366 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.314 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.315 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.316 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.317 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_stat 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.318 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_stat' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.319 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.320 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.321 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.322 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.323 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.324 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.325 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38277/exe 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.326 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.327 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.328 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_stat 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_stat 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.329 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38439 2024-04-29T20:24:50.330 INFO:tasks.workunit.client.0.smithi064.stdout:test api_stat on pid 38439 2024-04-29T20:24:50.337 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_stat on pid 38439' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38439 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.338 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_stat_pp 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_stat_pp' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_stat_pp 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.339 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.340 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.341 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38366/exe 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.342 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-29T20:24:50.343 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_stat_pp 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38475 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.344 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_stat_pp on pid 38475' 2024-04-29T20:24:50.345 INFO:tasks.workunit.client.0.smithi064.stdout:test api_stat_pp on pid 38475 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38475 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_watch_notify 2024-04-29T20:24:50.349 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_watch_notify' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_watch_notify 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.350 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.351 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_watch_notify 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.352 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38488 2024-04-29T20:24:50.353 INFO:tasks.workunit.client.0.smithi064.stdout:test api_watch_notify on pid 38488 2024-04-29T20:24:50.357 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_watch_notify on pid 38488' 2024-04-29T20:24:50.357 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38488 2024-04-29T20:24:50.357 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.357 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.357 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.357 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.358 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.359 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [----------] Global test environment set-up. 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.238+0000 7ff3e4845780 10 monclient: get_monmap_and_config 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.238+0000 7ff3e4845780 10 monclient: build_initial_monmap 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.238+0000 7ff3e4845780 10 monclient: monmap: 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: epoch 0 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:24:50.360 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: last_changed 0.000000 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: created 0.000000 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: min_mon_release 0 (unknown) 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: election_strategy: 1 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.239+0000 7ff3e4845780 1 Processor -- start 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.239+0000 7ff3e4845780 1 -- start start 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.239+0000 7ff3e4845780 10 monclient: init 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient: _reopen_session rank -1 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 0x5646090400d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): picked mon.noname-a con 0x564609007dc0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:24:50.361 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56460900c500 0x5646090cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): picked mon.noname-b con 0x56460900c500 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x5646090dea20 0x5646090cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): picked mon.noname-c con 0x5646090dea20 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): start opening mon connection 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5646090081a0 con 0x564609007dc0 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): start opening mon connection 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x564608fb9d70 con 0x56460900c500 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): start opening mon connection 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x5646090088a0 con 0x5646090dea20 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): _renew_subs 2024-04-29T20:24:50.362 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.240+0000 7ff3e4845780 10 monclient(hunting): authenticate will time out at 1639.399770s 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.375 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38334/exe 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.376 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.377 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.378 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.379 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.380 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_watch_notify_pp 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_watch_notify_pp' 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38354/exe 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.381 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.382 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.383 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_watch_notify_pp 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.384 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.385 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.386 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.386 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38439/exe 2024-04-29T20:24:50.386 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_watch_notify_pp 2024-04-29T20:24:50.386 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38532 2024-04-29T20:24:50.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-2test api_watch_notify_pp on pid 38532 2024-04-29T20:24:50.388 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_watch_notify_pp on pid 38532' 2024-04-29T20:24:50.388 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38532 2024-04-29T20:24:50.388 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.388 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.388 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_cmd 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_cmd' 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_cmd 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.389 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_cmd 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38542 2024-04-29T20:24:50.390 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_cmd on pid 38542' 2024-04-29T20:24:50.391 INFO:tasks.workunit.client.0.smithi064.stdout:test api_cmd on pid 38542 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38542 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_cmd_pp 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_cmd_pp' 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_cmd_pp 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.393 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-29T20:24:50.394 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_cmd_pp 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38549 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_cmd_pp on pid 38549' 2024-04-29T20:24:50.395 INFO:tasks.workunit.client.0.smithi064.stdout:test api_cmd_pp on pid 38549 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38549 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_stat: /' 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-29T20:24:50.400 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_pool: /' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.401 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.402 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_service 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_service' 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.403 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_service 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stdout:9T20:24:50.256+0000 7ff3e1b88640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 0x5646090400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.256+0000 7ff3e1b88640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 0x5646090400d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55838/0 (socket says 172.21.15.64:55838) 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.256+0000 7ff3e1b88640 1 -- 172.21.15.64:0/1799947664 learned_addr learned my addr 172.21.15.64:0/1799947664 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:24:50.404 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.256+0000 7ff3e1b88640 10 monclient(hunting): get_auth_request con 0x564609007dc0 auth_method 0 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.256+0000 7ff3e1b88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.256+0000 7ff3e1b88640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.256+0000 7ff3e1b88640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.257+0000 7ff3e1b88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.257+0000 7ff3e1b88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.257+0000 7ff3e1b88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 10 monclient(hunting): handle_auth_done global_id 4431 payload 274 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 1 -- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x5646090dea20 msgr2=0x5646090cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x5646090dea20 0x5646090cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 1 -- 172.21.15.64:0/1799947664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56460900c500 msgr2=0x5646090cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56460900c500 0x5646090cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 10 monclient: _finish_hunting 0 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.259+0000 7ff3e1b88640 1 monclient: found mon.noname-a 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.260+0000 7ff3e1b88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:24:50.405 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.260+0000 7ff3e1b88640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.260+0000 7ff3e1b88640 1 -- 172.21.15.64:0/1799947664 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564609041280 con 0x564609007dc0 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.272+0000 7ff3e1b88640 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 0x5646090400d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00a520 tx=0x7ff3dc00a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4fb014f4255f7e server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.293+0000 7ff3d3fff640 1 -- 172.21.15.64:0/1799947664 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00f5b0 con 0x564609007dc0 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: dump: 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: epoch 1 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: min_mon_release 19 (squid) 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: election_strategy: 1 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:24:50.406 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: _finish_auth 0 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: _check_auth_tickets 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 1 -- 172.21.15.64:0/1799947664 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3dc00f750 con 0x564609007dc0 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 1 -- 172.21.15.64:0/1799947664 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00fb40 con 0x564609007dc0 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3d3fff640 10 monclient: dump: 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: epoch 1 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:24:50.407 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: min_mon_release 19 (squid) 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: election_strategy: 1 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3e4845780 5 monclient: authenticate success, global_id 4431 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3e4845780 10 monclient: get_monmap_and_config success 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.294+0000 7ff3e4845780 4 set_mon_vals no callback set 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 10 monclient: shutdown 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 -- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 msgr2=0x5646090400d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:24:50.408 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 0x5646090400d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00a520 tx=0x7ff3dc00a9e0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-29T20:24:50.410 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 -- 172.21.15.64:0/1799947664 shutdown_connections 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x5646090dea20 0x5646090cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56460900c500 0x5646090cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 --2- 172.21.15.64:0/1799947664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564609007dc0 0x5646090400d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 -- 172.21.15.64:0/1799947664 >> 172.21.15.64:0/1799947664 conn(0x56460904a3f0 msgr2=0x564609044df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 -- 172.21.15.64:0/1799947664 shutdown_connections 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 1 -- 172.21.15.64:0/1799947664 wait complete. 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 10 monclient: build_initial_monmap 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.295+0000 7ff3e4845780 10 monclient: monmap: 2024-04-29T20:24:50.411 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: epoch 0 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: last_changed 0.000000 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: created 0.000000 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: min_mon_release 0 (unknown) 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: election_strategy: 1 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 1 librados: starting msgr at 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 1 librados: starting objecter 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 1 Processor -- start 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 1 -- start start 2024-04-29T20:24:50.412 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 1 librados: setting wanted keys 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 1 librados: calling monclient init 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.296+0000 7ff3e4845780 10 monclient: init 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient: _reopen_session rank -1 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x564609007dc0 0x564609043950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): picked mon.noname-b con 0x564609007dc0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x56460900c500 0x5646090cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.413 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): picked mon.noname-c con 0x56460900c500 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.419 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.420 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_service 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38581 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.421 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.422 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.423 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5646090dea20 0x5646090ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 ttest api_service on pid 38581 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_service on pid 38581' 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38581 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_service_pp 2024-04-29T20:24:50.425 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_service_pp' 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout:x=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): picked mon.noname-a con 0x5646090dea20 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): start opening mon connection 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56460900bb50 con 0x5646090dea20 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): start opening mon connection 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5646090418a0 con 0x564609007dc0 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): start opening mon connection 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x56460900bcd0 con 0x56460900c500 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): _renew_subs 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.297+0000 7ff3e4845780 10 monclient(hunting): authenticate will time out at 1639.457488s 2024-04-29T20:24:50.426 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.298+0000 7ff3e1387640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x56460900c500 0x5646090cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.299+0000 7ff3e1387640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x56460900c500 0x5646090cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:44488/0 (socket says 172.21.15.64:44488) 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.299+0000 7ff3e1387640 1 -- 172.21.15.64:0/171168981 learned_addr learned my addr 172.21.15.64:0/171168981 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.299+0000 7ff3e1387640 10 monclient(hunting): get_auth_request con 0x56460900c500 auth_method 0 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.299+0000 7ff3e1387640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.299+0000 7ff3e1387640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.299+0000 7ff3e1387640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.300+0000 7ff3e1387640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.300+0000 7ff3e1387640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.300+0000 7ff3e1387640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 1 -- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x564609007dc0 msgr2=0x564609043950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x564609007dc0 0x564609043950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.427 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 1 -- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5646090dea20 msgr2=0x5646090ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.436 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.437 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.438 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.439 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.440 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.441 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.442 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_service_pp 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.443 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_service_pp 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38612 2024-04-29T20:24:50.444 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04test api_service_pp on pid 38612 2024-04-29T20:24:50.453 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_service_pp on pid 38612' 2024-04-29T20:24:50.453 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38612 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_c_write_operations 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_c_write_operations' 2024-04-29T20:24:50.454 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38475/exe 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_c_write_operations 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.455 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38488/exe 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.456 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.457 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.458 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.459 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.460 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_c_write_operations 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.461 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38641 2024-04-29T20:24:50.462 INFO:tasks.workunit.client.0.smithi064.stdout:test api_c_write_operations on pid 38641 2024-04-29T20:24:50.470 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_c_write_operations on pid 38641' 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38641 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.471 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_c_read_operations 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_c_read_operations' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-29T20:24:50.472 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.473 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_c_read_operations 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38542/exe 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.474 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.475 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.476 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_cmd: /' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.477 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.478 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.479 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.480 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.481 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_c_read_operations 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38684 2024-04-29T20:24:50.482 INFO:tasks.workunit.client.0.smithi064.stdout:test api_c_read_operations on pid 38684 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_c_read_operations on pid 38684' 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38684 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.492 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.493 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.494 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.495 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.496 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.497 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.498 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s api_cls_remote_reads 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.499 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.500 INFO:tasks.workunit.client.0.smithi064.stdout:-29T20:24:50.308+0000 7ff3e1387640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5646090dea20 0x5646090ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 10 monclient: _finish_hunting 0 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 1 monclient: found mon.noname-c 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.308+0000 7ff3e1387640 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646090d6660 con 0x56460900c500 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.315+0000 7ff3e1387640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x56460900c500 0x5646090cc640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800a4f0 tx=0x7ff3d8053540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.315+0000 7ff3d2ffd640 10 client.?.objecter ms_handle_connect 0x56460900c500 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.315+0000 7ff3d2ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8062020 con 0x56460900c500 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: dump: 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: epoch 1 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:24:50.501 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: min_mon_release 19 (squid) 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: election_strategy: 1 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: _finish_auth 0 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d8053970 con 0x56460900c500 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8057040 con 0x56460900c500 2024-04-29T20:24:50.502 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:24:50.503 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:24:50.503 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d2ffd640 10 monclient: dump: 2024-04-29T20:24:50.503 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: epoch 1 2024-04-29T20:24:50.503 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:24:50.511 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' api_cls_remote_reads' 2024-04-29T20:24:50.511 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.511 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo api_cls_remote_reads 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.512 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.513 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.514 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.515 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38549/exe 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.516 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.517 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.518 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=api_cls_remote_reads 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38743 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test api_cls_remote_reads on pid 38743' 2024-04-29T20:24:50.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: last_changed 2024-04-29T20:21:08.test api_cls_remote_reads on pid 38743 2024-04-29T20:24:50.523 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38743 2024-04-29T20:24:50.523 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38641/exe 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.524 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s list_parallel 2024-04-29T20:24:50.525 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' list_parallel' 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo list_parallel 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38532/exe 2024-04-29T20:24:50.526 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.527 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=list_parallel 2024-04-29T20:24:50.527 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.527 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.527 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38755 2024-04-29T20:24:50.527 INFO:tasks.workunit.client.0.smithi064.stdout:test list_parallel on pid 38755 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test list_parallel on pid 38755' 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38755 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.531 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s open_pools_parallel 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' open_pools_parallel' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-29T20:24:50.532 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo open_pools_parallel 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.533 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=open_pools_parallel 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38581/exe 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38770 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test open_pools_parallel on pid 38770' 2024-04-29T20:24:50.534 INFO:tasks.workunit.client.0.smithi064.stdout:test open_pools_parallel on pid 38770 2024-04-29T20:24:50.558 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.558 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38770 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.561 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.562 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.563 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.564 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.565 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.566 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s delete_pools_parallel 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' delete_pools_parallel' 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.567 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.568 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.569 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.570 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_service.log 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_service: /' 2024-04-29T20:24:50.571 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.572 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.573 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.574 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.575 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.576 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.577 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.578 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38612/exe 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.579 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.580 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38743/exe 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.581 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.582 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38684/exe 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.583 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo delete_pools_parallel 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=delete_pools_parallel 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38853 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test delete_pools_parallel on pid 38853' 2024-04-29T20:24:50.584 INFO:tasks.workunit.client.0.smithi064.stdout:test delete_pools_parallel on pid 38853 2024-04-29T20:24:50.596 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38853 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s cls 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' cls' 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.597 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.598 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.599 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.600 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.601 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.602 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.603 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.604 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.605 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38770/exe 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo cls 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.606 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=cls 2024-04-29T20:24:50.607 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38897 2024-04-29T20:24:50.607 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test cls on pid 38897' 2024-04-29T20:24:50.607 INFO:tasks.workunit.client.0.smithi064.stdout:test cls on pid 38897 2024-04-29T20:24:50.613 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38897 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.614 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.615 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.616 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38853/exe 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.617 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.618 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s cmd 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' cmd' 2024-04-29T20:24:50.619 INFO:tasks.workunit.client.0.smithi064.stdout:698911+0000 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: min_mon_release 19 (squid) 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: election_strategy: 1 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.361+0000 7ff3d3fff640 4 set_mon_vals no callback set 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 5 monclient: authenticate success, global_id 4316 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 10 monclient: _renew_subs 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x5646090d6940 con 0x56460900c500 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:24:50.620 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 10 monclient: _renew_subs 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x564609043e90 con 0x56460900c500 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 1 librados: init done 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.362+0000 7ff3e4845780 10 librados: wait_for_osdmap waiting 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.450+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff3d8065020 con 0x56460900c500 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.451+0000 7ff3d2ffd640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff3c406c520 0x7ff3c406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.452+0000 7ff3e1b88640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff3c406c520 0x7ff3c406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.452+0000 7ff3e1b88640 10 monclient: get_auth_request con 0x7ff3c406c520 auth_method 0 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.452+0000 7ff3e1b88640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff3c406c520 0x7ff3c406e9e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00a520 tx=0x7ff3dc00af90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.452+0000 7ff3d2ffd640 10 client.4316.objecter ms_handle_connect 0x7ff3c406c520 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.461+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5210+0+0 (secure 0 0 0) 0x7ff3d8098350 con 0x56460900c500 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.461+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(27..27 src has 1..27) 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.461+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map got epochs [27,27] > 0 2024-04-29T20:24:50.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.461+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding full epoch 27 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.627 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.628 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38755/exe 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.629 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo cmd 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=cmd 2024-04-29T20:24:50.630 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.631 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.631 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.631 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38940 2024-04-29T20:24:50.631 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.631 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio:test cmd on pid 38940 2024-04-29T20:24:50.644 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test cmd on pid 38940' 2024-04-29T20:24:50.644 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=38940 2024-04-29T20:24:50.644 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.644 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.644 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.644 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ list_parallel: /' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s handler_error 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' handler_error' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.645 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.646 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.647 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.648 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.649 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.650 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.651 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo handler_error 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.652 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.653 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.654 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=handler_error 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39005 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test handler_error on pid 39005' 2024-04-29T20:24:50.655 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-29T20:24:50.656 INFO:tasks.workunit.client.0.smithi064.stdout:test handler_error on pid 39005 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39005 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s io 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' io' 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38897/exe 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.661 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.662 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.663 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_cls 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo io 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.664 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=io 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39025 2024-04-29T20:24:50.665 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test io on pid 39025' 2024-04-29T20:24:50.666 INFO:tasks.workunit.client.0.smithi064.stdout:test io on pid 39025 2024-04-29T20:24:50.668 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39025 2024-04-29T20:24:50.668 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.668 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.668 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-29T20:24:50.668 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s ec_io 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' ec_io' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo ec_io 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.669 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=ec_io 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39035 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test ec_io on pid 39035' 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.670 INFO:tasks.workunit.client.0.smithi064.stdout:test ec_io on pid 39035 2024-04-29T20:24:50.675 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39035 2024-04-29T20:24:50.675 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s list 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' list' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.676 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.677 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo list 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.678 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_cls.log 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ cls: /' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=list 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39060 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test list on pid 39060' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-29T20:24:50.679 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.680 INFO:tasks.workunit.client.0.smithi064.stdout:test list on pid 39060 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39060 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.695 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s ec_list 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' ec_list' 2024-04-29T20:24:50.696 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/38940/exe 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo ec_list 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.697 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.698 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_cmd 2024-04-29T20:24:50.699 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.700 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.701 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.702 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39005/exe 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.703 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ cmd: /' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.704 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_handler_error 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.705 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ handler_error: /' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.706 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.707 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=ec_list 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39132 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test ec_list on pid 39132' 2024-04-29T20:24:50.708 INFO:tasks.workunit.client.0.smithi064.stdout:test ec_list on pid 39132 2024-04-29T20:24:50.714 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39132 2024-04-29T20:24:50.714 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.714 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.714 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-29T20:24:50.714 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.715 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s misc 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' misc' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.716 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo misc 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.717 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.718 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=misc 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39173 2024-04-29T20:24:50.719 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test misc on pid 39173' 2024-04-29T20:24:50.720 INFO:tasks.workunit.client.0.smithi064.stdout:test misc on pid 39173 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39173 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s pool 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' pool' 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.729 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.730 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo pool 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.731 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.732 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.733 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39035/exe 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.734 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.735 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.736 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39060/exe 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.737 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.738 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.738 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-29T20:24:50.738 INFO:tasks.workunit.client.0.smithi064.stdout:test pool on pid 39194 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=pool 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.745 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39194 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test pool on pid 39194' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39194 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_ec_io 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.746 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.747 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_list.log 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s read_operations 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' read_operations' 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_list 2024-04-29T20:24:50.748 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.749 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39025/exe 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ ec_io: /' 2024-04-29T20:24:50.750 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo read_operations 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ list: /' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.751 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=read_operations 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.752 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39218 2024-04-29T20:24:50.753 INFO:tasks.workunit.client.0.smithi064.stdout:test read_operations on pid 39218 2024-04-29T20:24:50.759 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test read_operations on pid 39218' 2024-04-29T20:24:50.759 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39218 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s snapshots 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' snapshots' 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ io: /' 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.760 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_io.log 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_io 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.761 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo snapshots 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.762 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.763 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.763 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.763 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.767 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.768 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=snapshots 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39237 2024-04-29T20:24:50.769 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test snapshots on pid 39237' 2024-04-29T20:24:50.770 INFO:tasks.workunit.client.0.smithi064.stdout:test snapshots on pid 39237 2024-04-29T20:24:50.788 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39237 2024-04-29T20:24:50.788 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.788 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.788 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.788 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-29T20:24:50.788 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.789 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.790 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.791 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39132/exe 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.792 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.793 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s watch_notify 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' watch_notify' 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.794 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.795 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.796 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ ec_list: /' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.797 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_ec_list 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo watch_notify 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.798 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.799 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39173/exe 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.800 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_misc 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.801 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_misc.log 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.802 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.803 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ misc: /' 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=watch_notify 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39284 2024-04-29T20:24:50.804 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test watch_notify on pid 39284' 2024-04-29T20:24:50.805 INFO:tasks.workunit.client.0.smithi064.stdout:test watch_notify on pid 39284 2024-04-29T20:24:50.809 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39284 2024-04-29T20:24:50.809 INFO:tasks.workunit.client.0.smithi064.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-29T20:24:50.809 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.809 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:++ printf %25s write_operations 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+ r=' write_operations' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.810 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.811 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++ echo write_operations 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++ awk '{print $1}' 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39194/exe 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+ ff=write_operations 2024-04-29T20:24:50.812 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39310 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.813 INFO:tasks.workunit.client.0.smithi064.stdout:test write_operations on pid 39310 2024-04-29T20:24:50.838 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.838 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'test write_operations on pid 39310' 2024-04-29T20:24:50.838 INFO:tasks.workunit.client.0.smithi064.stderr:+ pids[$f]=39310 2024-04-29T20:24:50.838 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.838 INFO:tasks.workunit.client.0.smithi064.stderr:+ ret=0 2024-04-29T20:24:50.838 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' 1 -eq 1 ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38277 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38277 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.839 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.840 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -f /etc/bashrc ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:+ . /etc/bashrc 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -z '' ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ BASHRCSOURCED=Y 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ shopt -q login_shell 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ pool: /' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.841 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_pool 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_pool.log 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.842 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:+++ umask 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' 0022 -eq 0 ']' 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ SHELL=/bin/bash 2024-04-29T20:24:50.843 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-29T20:24:50.844 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39218/exe 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.845 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.846 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_read_operations 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'grep=grep --color=auto' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-29T20:24:50.847 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorls.sh 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' '!' -t 0 ']' 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:+++ return 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export S_COLORS=auto 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:+++ S_COLORS=auto 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-29T20:24:50.848 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ read_operations: /' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.849 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.850 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-29T20:24:50.851 INFO:tasks.workunit.client.0.smithi064.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39237/exe 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.852 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.853 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/gawk.sh 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-29T20:24:50.854 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/lang.sh 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ snapshots: /' 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/locale 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_snapshots 2024-04-29T20:24:50.855 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.856 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39310/exe 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.857 INFO:tasks.workunit.client.0.smithi064.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:++++ export LANG=en_US.UTF-8 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:++++ LANG=en_US.UTF-8 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ LANG=en_US.UTF-8 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+++ unset LANG_backup config 2024-04-29T20:24:50.858 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n '' ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' dumb = linux ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_write_operations 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/less.sh 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ for i in /etc/profile.d/*.sh 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ '[' '' ']' 2024-04-29T20:24:50.859 INFO:tasks.workunit.client.0.smithi064.stderr:++ . /etc/profile.d/which2.sh 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ write_operations: /' 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+++++ readlink /proc/39284/exe 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:++++ basename /usr/bin/bash 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_declare='declare -f' 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+++ which_opt=-f 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export which_declare 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+++ export -f which 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset i 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:++ unset -f pathmunge 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+ export PATH 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-29T20:24:50.860 INFO:tasks.workunit.client.0.smithi064.stderr:+ unset rc 2024-04-29T20:24:50.861 INFO:tasks.workunit.client.0.smithi064.stderr:+ ceph_test_neorados_watch_notify 2024-04-29T20:24:50.861 INFO:tasks.workunit.client.0.smithi064.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-29T20:24:50.861 INFO:tasks.workunit.client.0.smithi064.stderr:+ sed 's/^/ watch_notify: /' 2024-04-29T20:24:52.034 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:10.540 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.044 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:10.550 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.064 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:10.570 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.081 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:10.587 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.110 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:10.587 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.120 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:10.597 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.145 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:10.622 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.146 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:10.652 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:52.181 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:10.687 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:24:54.056 INFO:tasks.workunit.client.0.smithi064.stdout: 2024-04-29T20:24:50.461+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:24:54.056 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.462+0000 7ff3e4845780 10 librados: wait_for_osdmap done waiting 2024-04-29T20:24:54.056 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.462+0000 7ff3e4845780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.462+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff3dc00f750 con 0x56460900c500 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff3d8096550 con 0x56460900c500 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3d2ffd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3d37fe640 10 client.4316.objecter _get_latest_version latest 27, have it 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3e4845780 10 client.4316.objecter create_pool name=ceph_test_rados_api_asio 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3e4845780 10 client.4316.objecter pool_op_submit 1 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3e4845780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:50.529+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x564608fc0700 con 0x56460900c500 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.297+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.297+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.297+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.297+0000 7ff3d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v29) ==== 43+0+0 (secure 0 0 0) 0x7ff3d806c030 con 0x56460900c500 2024-04-29T20:24:54.057 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 pool_op_reply(tid 1 (0) Success v29) 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 client.4316.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v29) 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 client.4316.objecter have request 1 at 0x5646090d3da0 Op: create 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 20 client.4316.objecter waiting for client to reach epoch 29 before calling back 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 monclient: _renew_subs 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=28}) -- 0x7ff3c40722c0 con 0x56460900c500 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:53.589+0000 7ff3d2ffd640 10 client.4316.objecter done 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.054+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 8 ==== osd_map(28..29 src has 1..29) ==== 15896+0+0 (secure 0 0 0) 0x7ff3d80985b0 con 0x56460900c500 2024-04-29T20:24:54.058 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(28..29 src has 1..29) 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map got epochs [28,29] > 27 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 28 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 29 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 10 librados: set snap read head -> 1 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 10 librados: writefull oid=exist nspace= 2024-04-29T20:24:56.053 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 29 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x5646090d49a0 0x56460918b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 2 osd.0 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 2 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 15 client.4316.objecter _send_op 2 to 11.1 on osd.0 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:2 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56460918b720 con 0x5646090d49a0 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.055+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:24:56.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:54.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:55.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:55.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:55.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:55.363+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:55.996+0000 7ff3e2389640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x5646090d49a0 0x56460918b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:24:56.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:56.020+0000 7ff3e2389640 10 monclient: get_auth_request con 0x5646090d49a0 auth_method 0 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 20 api_service: [==========] Running 4 tests from 1 test suite. 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [----------] Global test environment set-up. 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5921 ms) 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ OK ] LibRadosService.RegisterLate (640 ms) 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: cluster: 2024-04-29T20:25:02.671 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: id: 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: health: HEALTH_WARN 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 26 pool(s) do not have an application enabled 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: services: 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 3m) 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: data: 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: pools: 37 pools, 1129 pgs 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: objects: 2 objects, 449 KiB 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: pgs: 97.609% pgs unknown 2024-04-29T20:25:02.672 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 1.594% pgs not active 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 1102 unknown 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 18 creating+peering 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 9 active+clean 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: cluster: 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: id: 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: health: HEALTH_WARN 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 17 pool(s) do not have an application enabled 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: services: 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 3m) 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: mgr: x(active, since 3m), standbys: y 2024-04-29T20:25:02.673 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: data: 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: pools: 36 pools, 1049 pgs 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: objects: 3 objects, 449 KiB 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: pgs: 54.528% pgs unknown 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 44.137% pgs not active 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 572 unknown 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 442 creating+peering 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 21 creating+activating 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 14 active+clean 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.674 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: io: 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: client: 183 B/s wr, 0 op/s rd, 0 op/s wr 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: cluster: 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: id: 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: health: HEALTH_WARN 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 21 pool(s) do not have an application enabled 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: services: 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 3m) 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: mgr: x(active, since 3m), standbys: y 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-29T20:25:02.675 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-29T20:25:02.676 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:02.676 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: data: 2024-04-29T20:25:02.676 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: pools: 40 pools, 1177 pgs 2024-04-29T20:25:05.366 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: objects: 3 objects, 449 KiB24-04-29T20:24:56.051+0000 7ff3d2ffd640 10 client.4316.objecter ms_handle_connect 0x5646090d49a0 2024-04-29T20:25:05.366 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:56.051+0000 7ff3e2389640 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x5646090d49a0 0x56460918b1e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:56.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:56.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:56.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:57.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:57.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:57.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:58.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:58.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:58.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:59.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:59.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:24:59.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:00.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.367 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:00.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:00.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:00.363+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:01.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:01.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:01.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:02.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:02.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:02.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:03.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:03.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:03.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:03.298+0000 7ff3d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:04.298+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:04.298+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.368 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:04.298+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.369 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:05.369 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:05.369 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:05.369 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 2 client.4316.objecter tid 2 on osd.0 is laggy 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 10 monclient: _renew_subs 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=30}) -- 0x7ff3b4002f90 con 0x56460900c500 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.364+0000 7ff3e3dec640 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- ping magic: 0 -- 0x7ff3b4003360 con 0x5646090d49a0 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 9 ==== osd_map(30..36 src has 1..36) ==== 19283+0+0 (secure 0 0 0) 0x7ff3d809a240 con 0x56460900c500 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(30..36 src has 1..36) 2024-04-29T20:25:05.446 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map got epochs [30,36] > 29 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 30 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 30 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 31 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 31 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 32 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 32 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 33 2024-04-29T20:25:05.447 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:05.448 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 33 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:05.448 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:05.448 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 34 2024-04-29T20:25:05.448 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:05.448 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 34 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 35 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 35 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 36 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:10.371 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 36 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:05.443+0000 7ff3d2ffd640 20 client.4316.objecter 2 11.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:06.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:06.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:06.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:07.299+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:07.299+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:07.299+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:08.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:08.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:10.372 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:08.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:09.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:09.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:09.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 2 client.4316.objecter tid 2 on osd.0 is laggy 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 10 monclient: _renew_subs 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:25:10.373 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=37}) -- 0x7ff3b4002f90 con 0x56460900c500 2024-04-29T20:25:10.374 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.365+0000 7ff3e3dec640 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- ping magic: 0 -- 0x7ff3b40035c0 con 0x5646090d49a0 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== mon.2 v2:172.21.15.64:3301/0 10 ==== osd_map(37..38 src has 1..38) ==== 1664+0+0 (secure 0 0 0) 0x7ff3d8096920 con 0x56460900c500 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(37..38 src has 1..38) 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map got epochs [37,38] > 36 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 37 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 37 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 38 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 10 client.4316.objecter checking op 2 2024-04-29T20:25:13.516 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:10.369+0000 7ff3d2ffd640 20 client.4316.objecter 2 11.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.835+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 1 ==== osd_map(30..36 src has 1..36) ==== 19283+0+0 (crc 0 0 0) 0x7ff3d400a500 con 0x5646090d49a0 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.835+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(30..36 src has 1..36) 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.835+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map ignoring epochs [30,36] <= 38 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.835+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:11.835+0000 7ff3d2ffd640 20 client.4316.objecter 2 11.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:12.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:12.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:12.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:13.517 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.300+0000 7ff3d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 2 ==== osd_op_reply(2 exist [writefull 0~5] v36'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(2 exist [writefull 0~5] v36'1 uv1 ondisk = 0) 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 2 ondisk uv 1 in 11.1 attempt 0 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 2 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 15 client.4316.objecter _finish_op 2 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 2 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:13.518 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 librados: Objecter returned from writefull r=0 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [read 0~256] tid 3 osd.0 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 3 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 15 client.4316.objecter _send_op 3 to 11.1 on osd.0 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:3 11.1 11:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) -- 0x564609181ea0 con 0x5646090d49a0 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x564609182240 2024-04-29T20:25:13.519 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:13.520 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:13.520 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:13.520 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.520 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid noexist '@11' '@11' [read 0~256] tid 4 osd.0 2024-04-29T20:25:13.520 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.577 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 4 2024-04-29T20:25:13.577 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 15 client.4316.objecter _send_op 4 to 11.1 on osd.0 2024-04-29T20:25:13.577 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:4 11.1 11:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) -- 0x5646091826f0 con 0x5646090d49a0 2024-04-29T20:25:13.577 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.514+0000 7ff3e4845780 5 client.4316.objecter 2 in flight 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 3 ondisk uv 1 in 11.1 attempt 0 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 5 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 3 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 15 client.4316.objecter _finish_op 3 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 3 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 5 client.4316.objecter 1 in flight 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.575+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-29T20:25:13.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 4 ondisk uv 0 in 11.1 attempt 0 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 10 client.4316.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 4 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 15 client.4316.objecter _finish_op 4 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 4 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (61 ms) 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-29T20:25:13.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x564609182240 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [read 0~256] tid 5 osd.0 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:13.613 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 5 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 15 client.4316.objecter _send_op 5 to 11.1 on osd.0 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:5 11.1 11:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) -- 0x564609182ac0 con 0x5646090d49a0 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid noexist '@11' '@11' [read 0~256] tid 6 osd.0 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 6 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 15 client.4316.objecter _send_op 6 to 11.1 on osd.0 2024-04-29T20:25:13.614 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:6 11.1 11:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) -- 0x56460918bac0 con 0x5646090d49a0 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.576+0000 7ff3e4845780 5 client.4316.objecter 2 in flight 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 5 ondisk uv 1 in 11.1 attempt 0 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 5 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 5 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 15 client.4316.objecter _finish_op 5 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 5 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 5 client.4316.objecter 1 in flight 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.615 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 6 ondisk uv 0 in 11.1 attempt 0 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 6 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 15 client.4316.objecter _finish_op 6 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 6 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (36 ms) 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:13.616 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:13.617 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:13.617 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [read 0~256] tid 7 osd.0 2024-04-29T20:25:13.617 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:13.617 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 7 2024-04-29T20:25:13.617 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 15 client.4316.objecter _send_op 7 to 11.1 on osd.0 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:7 11.1 11:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) -- 0x564609182240 con 0x5646090d49a0 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646091cc4f0 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid noexist '@11' '@11' [read 0~256] tid 8 osd.0 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 8 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 15 client.4316.objecter _send_op 8 to 11.1 on osd.0 2024-04-29T20:25:13.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:8 11.1 11:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38) -- 0x5646091cc9a0 con 0x5646090d49a0 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.611+0000 7ff3e4845780 5 client.4316.objecter 2 in flight 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 7 ondisk uv 1 in 11.1 attempt 0 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 5 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 7 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 15 client.4316.objecter _finish_op 7 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 7 2024-04-29T20:25:13.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 5 client.4316.objecter 1 in flight 2024-04-29T20:25:13.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:13.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 8 ondisk uv 0 in 11.1 attempt 0 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 8 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 15 client.4316.objecter _finish_op 8 2024-04-29T20:25:14.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (20 ms) 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 8 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 librados: queue_aio_write 0x564608f7a5f0 completion 0x564608f56810 write_seq 1 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:14.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [write 0~5 in=5b] tid 9 osd.0 2024-04-29T20:25:14.533 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 9 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 15 client.4316.objecter _send_op 9 to 11.1 on osd.0 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:9 11.1 11:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) -- 0x56460918ce70 con 0x5646090d49a0 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:13.631+0000 7ff3e4845780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:14.534 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 9 ==== osd_op_reply(9 exist [write 0~5] v38'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:16.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(9 exist [write 0~5] v38'2 uv2 ondisk = 0) 2024-04-29T20:25:16.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:16.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 9 ondisk uv 2 in 11.1 attempt 0 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 9 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 15 client.4316.objecter _finish_op 9 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 9 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e2389640 20 librados: complete_aio_write 0x564608f56810 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (896 ms) 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 librados: queue_aio_write 0x564608f7a5f0 completion 0x564608f45b40 write_seq 2 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:16.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [write 0~5 in=5b] tid 10 osd.0 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 10 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 15 client.4316.objecter _send_op 10 to 11.1 on osd.0 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:10 11.1 11:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) -- 0x56460918ce70 con 0x5646090d49a0 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:14.527+0000 7ff3e4845780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:15.300+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:15.300+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:15.300+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:16.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:15.366+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:25:16.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:17.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:17.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:17.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 10 ==== osd_op_reply(10 exist [write 0~5] v38'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:17.302 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(10 exist [write 0~5] v38'3 uv3 ondisk = 0) 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 10 ondisk uv 3 in 11.1 attempt 0 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 10 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 15 client.4316.objecter _finish_op 10 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 10 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e2389640 20 librados: complete_aio_write 0x564608f45b40 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2366 ms) 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 librados: queue_aio_write 0x564608f7a5f0 completion 0x56460918d090 write_seq 3 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:17.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [write 0~5 in=5b] tid 11 osd.0 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 11 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 15 client.4316.objecter _send_op 11 to 11.1 on osd.0 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:11 11.1 11:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) -- 0x5646091af9f0 con 0x5646090d49a0 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:17.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:16.893+0000 7ff3e4845780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-29T20:25:18.402 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:36.908 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:18.523 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:37.029 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:18.587 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:37.093 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:18.591 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:37.097 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:18.681 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:37.158 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:18.752 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:37.229 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:18.964 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:37.470 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:19.568 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:38.073 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:20.603 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:39.080 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5391 ms) 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (81 ms) 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20310 ms) 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: attempt 0 of 20 2024-04-29T20:25:22.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5832 ms) 2024-04-29T20:25:22.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31614 ms total) 2024-04-29T20:25:22.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: 2024-04-29T20:25:22.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-29T20:25:22.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31614 ms total) 2024-04-29T20:25:22.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: pgs: 21.240% pgs unknown 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 76.805% pgs not active 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 854 creating+peering 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 250 unknown 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 50 creating+activating 2024-04-29T20:25:22.762 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 23 active+clean 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: io: 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: client: 170 B/s wr, 0 op/s rd, 0 op/s wr 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5693 ms) 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ OK ] LibRadosService.Status (20056 ms) 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [----------] 4 tests from LibRadosService (32310 ms total) 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [----------] Global test environment tear-down 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32310 ms total) 2024-04-29T20:25:22.763 INFO:tasks.workunit.client.0.smithi064.stdout: api_service: [ PASSED ] 4 tests. 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:17.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:17.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:17.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:18.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:18.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:18.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:19.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:19.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:19.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.367+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 11 ==== osd_map(39..39 src has 1..40) ==== 312+0+0 (crc 0 0 0) 0x7ff3d4005c50 con 0x5646090d49a0 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(39..39 src has 1..40) 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map got epochs [39,39] > 38 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 39 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 10 client.4316.objecter checking op 11 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 39 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:25:24.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:20.548+0000 7ff3d2ffd640 20 client.4316.objecter 11 11.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:21.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:21.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:21.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:22.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:22.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:22.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:23.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:23.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:23.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:23.301+0000 7ff3d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:24.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:24.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:24.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:24.426 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:25:24.393+0000 12a65640 -1 osd.0 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4509.0:2 20.1 20:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-04-29T20:25:24.632 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:25:24.588+0000 12a65640 -1 osd.1 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4400.0:2 31.15 31.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-04-29T20:25:24.731 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:25:24.713+0000 12a65640 -1 osd.6 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4385.0:2 30.1a 30:5c6776db:::0.obj:head [write 0~75 in=75b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:24.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 12 ==== osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 11 ondisk uv 4 in 11.1 attempt 0 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 11 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 15 client.4316.objecter _finish_op 11 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 11 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:25.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.109+0000 7ff3e2389640 20 librados: complete_aio_write 0x56460918d090 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (8216 ms) 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646090d3fe0 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 39 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [read 0~0] tid 12 osd.0 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 12 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 15 client.4316.objecter _send_op 12 to 11.1 on osd.0 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:12 11.1 11:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x56460918be60 con 0x5646090d49a0 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:25.114 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918c200 2024-04-29T20:25:25.114 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 39 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.114 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:25.437 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:25:25.431+0000 12a65640 -1 osd.0 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4509.0:2 20.1 20:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-04-29T20:25:25.538 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid noexist '@11' '@11' [read 0~0] tid 13 osd.0 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 13 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 15 client.4316.objecter _send_op 13 to 11.1 on osd.0 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:13 11.1 11:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x56460918c6b0 con 0x5646090d49a0 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.110+0000 7ff3e4845780 5 client.4316.objecter 2 in flight 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.368+0000 7ff3e3dec640 10 client.4316.objecter tick 2024-04-29T20:25:25.539 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3d2ffd640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 13 ==== osd_map(40..43 src has 1..43) ==== 2545+0+0 (crc 0 0 0) 0x7ff3d4005c50 con 0x5646090d49a0 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 14 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3d2ffd640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_map(40..43 src has 1..43) 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map got epochs [40,43] > 39 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.536+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 40 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 12 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 40 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 13 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 40 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 41 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 12 2024-04-29T20:25:25.540 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 41 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 13 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 41 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 42 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 12 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 42 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 13 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 42 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 3 client.4316.objecter handle_osd_map decoding incremental epoch 43 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 12 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 43 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.541 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 10 client.4316.objecter checking op 13 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target epoch 43 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter 12 11.d3a59201 osd.0 exist [read 0~0] 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3d2ffd640 20 client.4316.objecter 13 11.a7bbaa1 osd.0 noexist [read 0~0] 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 12 ondisk uv 4 in 11.1 attempt 0 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 5 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 12 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 15 client.4316.objecter _finish_op 12 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 12 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 5 client.4316.objecter 1 in flight 2024-04-29T20:25:25.542 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 15 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 13 ondisk uv 0 in 11.1 attempt 0 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 10 client.4316.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 13 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 15 client.4316.objecter _finish_op 13 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.543 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 13 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (428 ms) 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918be60 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [read 0~0] tid 14 osd.0 2024-04-29T20:25:25.544 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 14 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 15 client.4316.objecter _send_op 14 to 11.1 on osd.0 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:14 11.1 11:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x5646090d40f0 con 0x5646090d49a0 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918c4b0 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:25.545 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:25.546 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:25.745 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:25:25.710+0000 12a65640 -1 osd.1 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4400.0:2 31.15 31.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-04-29T20:25:27.117 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid noexist '@11' '@11' [read 0~0] tid 15 osd.0 2024-04-29T20:25:27.117 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 15 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 15 client.4316.objecter _send_op 15 to 11.1 on osd.0 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:15 11.1 11:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x56460918c960 con 0x5646090d49a0 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:25.537+0000 7ff3e4845780 5 client.4316.objecter 2 in flight 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:26.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:26.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:26.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1578 ms) 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 16 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:27.118 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 14 ondisk uv 4 in 11.1 attempt 0 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 5 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 14 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 15 client.4316.objecter _finish_op 14 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 14 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 5 client.4316.objecter 1 in flight 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 17 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 15 ondisk uv 0 in 11.1 attempt 0 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:27.119 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 15 2024-04-29T20:25:27.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 15 client.4316.objecter _finish_op 15 2024-04-29T20:25:27.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:27.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 15 2024-04-29T20:25:27.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:27.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918be60 2024-04-29T20:25:27.303 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [read 0~0] tid 16 osd.0 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 16 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 15 client.4316.objecter _send_op 16 to 11.1 on osd.0 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:16 11.1 11:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x5646090d40f0 con 0x5646090d49a0 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x5646091af4d0 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2024-04-29T20:25:27.304 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid noexist '@11' '@11' [read 0~0] tid 17 osd.0 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 17 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 15 client.4316.objecter _send_op 17 to 11.1 on osd.0 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:17 11.1 11:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x5646091af980 con 0x5646090d49a0 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:27.305 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.115+0000 7ff3e4845780 5 client.4316.objecter 2 in flight 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.301+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.301+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.301+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 18 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 16 ondisk uv 4 in 11.1 attempt 0 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 5 2024-04-29T20:25:27.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 16 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 15 client.4316.objecter _finish_op 16 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 5 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 16 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 5 client.4316.objecter 1 in flight 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 19 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 17 ondisk uv 0 in 11.1 attempt 0 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (610 ms) 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 17 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 15 client.4316.objecter _finish_op 17 2024-04-29T20:25:27.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:27.729 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 17 2024-04-29T20:25:27.729 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:27.729 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 20 librados: queue_aio_write 0x564608f7a5f0 completion 0x56460918c3a0 write_seq 4 2024-04-29T20:25:27.729 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918be60 2024-04-29T20:25:27.729 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:28.959 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 18 osd.0 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 18 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 15 client.4316.objecter _send_op 18 to 11.1 on osd.0 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:18 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) -- 0x5646090d40f0 con 0x5646090d49a0 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:27.725+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:28.960 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.302+0000 7ff3d1ffb640 10 monclient: tick 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.302+0000 7ff3d1ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.302+0000 7ff3d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 20 ==== osd_op_reply(18 exist [writefull 0~5] v43'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(18 exist [writefull 0~5] v43'5 uv5 ondisk = 0) 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 18 ondisk uv 5 in 11.1 attempt 0 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 18 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 15 client.4316.objecter _finish_op 18 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 18 2024-04-29T20:25:28.961 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e2389640 20 librados: complete_aio_write 0x56460918c3a0 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1233 ms) 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 20 librados: queue_aio_write 0x564608f7a5f0 completion 0x5646091aff60 write_seq 5 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918be60 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:28.962 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 19 osd.0 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 19 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 15 client.4316.objecter _send_op 19 to 11.1 on osd.0 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:19 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) -- 0x5646090d40f0 con 0x5646090d49a0 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:28.958+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 21 ==== osd_op_reply(19 exist [writefull 0~5] v43'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(19 exist [writefull 0~5] v43'6 uv6 ondisk = 0) 2024-04-29T20:25:29.064 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 19 ondisk uv 6 in 11.1 attempt 0 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 19 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 15 client.4316.objecter _finish_op 19 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 19 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e2389640 20 librados: complete_aio_write 0x5646091aff60 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (104 ms) 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 20 librados: queue_aio_write 0x564608f7a5f0 completion 0x5646091afcd0 write_seq 6 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 10 client.4316.objecter _op_submit op 0x56460918be60 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 20 client.4316.objecter _calc_target epoch 43 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 20 client.4316.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2024-04-29T20:25:29.065 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 10 client.4316.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2024-04-29T20:25:29.066 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 20 client.4316.objecter _get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:29.066 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 10 client.4316.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 20 osd.0 2024-04-29T20:25:29.111 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 20 client.4316.objecter get_session s=0x5646090d3b50 osd=0 3 2024-04-29T20:25:29.111 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 15 client.4316.objecter _session_op_assign 0 20 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 15 client.4316.objecter _send_op 20 to 11.1 on osd.0 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- osd_op(unknown.0.0:20 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) -- 0x5646090d40f0 con 0x5646090d49a0 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.062+0000 7ff3e4845780 5 client.4316.objecter 1 in flight 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 <== osd.0 v2:172.21.15.64:6808/2796311601 22 ==== osd_op_reply(20 exist [writefull 0~5] v43'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7ff3d400a980 con 0x5646090d49a0 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 10 client.4316.objecter ms_dispatch 0x564608fb1ac0 osd_op_reply(20 exist [writefull 0~5] v43'7 uv7 ondisk = 0) 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 10 client.4316.objecter in handle_osd_op_reply 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 7 client.4316.objecter handle_osd_op_reply 20 ondisk uv 7 in 11.1 attempt 0 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 10 client.4316.objecter op 0 rval 0 len 0 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 15 client.4316.objecter handle_osd_op_reply completed tid 20 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 15 client.4316.objecter _finish_op 20 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 4 2024-04-29T20:25:29.112 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 15 client.4316.objecter _session_op_remove 0 20 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 5 client.4316.objecter 0 in flight 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 20 librados: complete_aio_write 0x5646091afcd0 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (48 ms) 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 10 librados: watch_flush enter 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 10 librados: watch_flush exit 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 10 client.4316.objecter close_session for osd.0 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x5646090d49a0 msgr2=0x56460918b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x5646090d49a0 0x56460918b1e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 20 client.4316.objecter put_session s=0x5646090d3b50 osd=0 1 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 20 client.4316.objecter shutdown clearing up homeless session... 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 10 client.4316.objecter successfully canceled tick 2024-04-29T20:25:29.113 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff3c406c520 msgr2=0x7ff3c406e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff3c406c520 0x7ff3c406e9e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00a520 tx=0x7ff3dc00af90 comp rx=0 tx=0).stop 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 10 monclient: shutdown 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x56460900c500 msgr2=0x5646090cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 --2- 172.21.15.64:0/171168981 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x56460900c500 0x5646090cc640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800a4f0 tx=0x7ff3d8053540 comp rx=0 tx=0).stop 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e2389640 1 -- 172.21.15.64:0/171168981 reap_dead start 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 shutdown_connections 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 >> 172.21.15.64:0/171168981 conn(0x56460904a3f0 msgr2=0x564609040730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 shutdown_connections 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 -- 172.21.15.64:0/171168981 wait complete. 2024-04-29T20:25:29.115 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.110+0000 7ff3e4845780 1 librados: shutdown 2024-04-29T20:25:29.116 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [----------] 12 tests from AsioRados (15596 ms total) 2024-04-29T20:25:29.116 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: 2024-04-29T20:25:29.116 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [----------] Global test environment tear-down 2024-04-29T20:25:29.116 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (38872 ms total) 2024-04-29T20:25:29.116 INFO:tasks.workunit.client.0.smithi064.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-29T20:25:33.654 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:33.642+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:34.026 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-29T20:25:34.026 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] Global test environment set-up. 2024-04-29T20:25:34.026 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-29T20:25:34.026 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: 2024-04-29T20:24:51.398+0000 7fbdef158780 -1 failed for service _ceph-mon._tcp 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: 2024-04-29T20:24:51.398+0000 7fbdef158780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (1163 ms) 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2007 ms) 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (3170 ms total) 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: 2024-04-29T20:25:34.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575fea99220 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575ff494300 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575fe90fca0 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575fe7a1660 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3b8d0 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3c2c0 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3cd10 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3d710 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3e140 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3e7a0 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3edd0 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3f830 2024-04-29T20:25:34.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3de70 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb40320 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb40e20 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb3f550 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb41cd0 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb40b90 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb42c00 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb41b10 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb43b70 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb42910 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb446b0 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb45140 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb43870 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb45c50 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb466e0 2024-04-29T20:25:34.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb44ef0 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb475b0 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb46460 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb48110 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb48bc0 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb47320 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb49720 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb4a1e0 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb48940 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 0x5575feb4ad20 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: started 37 aios 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575fea99220 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575ff494300 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575fe90fca0 2024-04-29T20:25:34.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575fe7a1660 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3b8d0 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3c2c0 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3cd10 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3d710 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3e140 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3e7a0 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3edd0 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3f830 2024-04-29T20:25:34.031 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3de70 2024-04-29T20:25:34.647 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:34.640+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:35.687 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:35.682+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:36.246 INFO:tasks.ceph.mon.c.smithi064.stderr:2024-04-29T20:25:36.174+0000 ffb1640 -1 mon.c@2(peon).paxos(paxos updating c 1..261) lease_expire from mon.0 v2:172.21.15.64:3300/0 is 2.440195322s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-04-29T20:25:36.663 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:36.654+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:37.718 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:37.713+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:38.699 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:38.694+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:39.753 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:39.748+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:40.734 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:40.729+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:41.002 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5 api_list: [==========] Running 11 tests from 3 test suites. 2024-04-29T20:25:41.002 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] Global test environment set-up. 2024-04-29T20:25:41.002 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-29T20:25:41.002 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-29T20:25:41.002 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.ListObjects (34998 ms) 2024-04-29T20:25:41.002 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1496 ms) 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset foo1,foo2,foo3 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo1 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo2 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo3 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset foo1,foo4,foo5 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo4 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo5 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo1 2024-04-29T20:25:41.003 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset foo6,foo7 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo7 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo6 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns1:foo4 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns1:foo5 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns2:foo7 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns2:foo6 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns1:foo1 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: :foo1 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: :foo2 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: :foo3 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (2266 ms) 2024-04-29T20:25:41.004 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 1 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 10 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 13 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 7 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 14 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 0 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 15 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 11 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 5 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 8 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 6 0 2024-04-29T20:25:41.005 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 3 0 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 4 0 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 12 0 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 9 0 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 2 0 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (2788 ms) 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: x cursor=MIN 2024-04-29T20:25:41.006 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2024-04-29T20:25:41.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:566253c9:::13:head 2024-04-29T20:25:41.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-04-29T20:25:41.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2024-04-29T20:25:41.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-04-29T20:25:41.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-04-29T20:25:41.704 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:41.697+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:42.722 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:42.715+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:863748b0:::15:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2024-04-29T20:25:42.859 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2024-04-29T20:25:42.860 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:e960b815:::9:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2024-04-29T20:25:42.861 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=3 expected=3 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=6 expected=6 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=8 expected=8 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-04-29T20:25:42.862 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=5 expected=5 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=11 expected=11 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:863748b0:::15:head 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=15 expected=15 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=0 expected=0 2024-04-29T20:25:42.863 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-04-29T20:25:42.864 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2024-04-29T20:25:42.864 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-04-29T20:25:42.864 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=14 expected=14 2024-04-29T20:25:42.864 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-04-29T20:25:44.075 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:43.967+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:44.988 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:44.960+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:46.186 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:46.177+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:47.205 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:47.199+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:48.186 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:48.175+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:48.417 INFO:tasks.ceph.mon.c.smithi064.stderr:2024-04-29T20:25:48.327+0000 ffb1640 -1 mon.c@2(peon).paxos(paxos updating c 1..263) lease_expire from mon.0 v2:172.21.15.64:3300/0 is 1.346266270s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-04-29T20:25:49.206 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:49.201+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:50.236 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:50.231+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:51.248 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:51.241+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:52.256 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:52.243+0000 ea5d640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:52.423 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-29T20:25:52.423 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (61946 ms) 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (61946 ms total) 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (61947 ms total) 2024-04-29T20:25:52.424 INFO:tasks.workunit.client.0.smithi064.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-29T20:25:53.591 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:53.294+0000 ea5d640 -1 osd.2 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:53.705 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (30562 ms) 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (7619 ms) 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-29T20:25:53.706 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (238 ms) 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (709 ms) 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (225 ms) 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (191 ms) 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (211 ms) 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (249 ms) 2024-04-29T20:25:53.707 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (141 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (345 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (156 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (109 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (365 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (276 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (751 ms) 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-29T20:25:53.708 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (227 ms) 2024-04-29T20:25:53.709 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (42374 ms total) 2024-04-29T20:25:53.709 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: 2024-04-29T20:25:53.709 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-29T20:25:53.709 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (63237 ms total) 2024-04-29T20:25:53.709 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-29T20:25:54.725 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: Running main() from gmock_main.cc 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [----------] Global test environment set-up. 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (64137 ms) 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [----------] 1 test from neocls_handler_error (64137 ms total) 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [----------] Global test environment tear-down 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [==========] 1 test from 1 test suite ran. (64137 ms total) 2024-04-29T20:25:54.726 INFO:tasks.workunit.client.0.smithi064.stdout: handler_error: [ PASSED ] 1 test. 2024-04-29T20:25:54.885 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:54.680+0000 ea5d640 -1 osd.2 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (963 ms) 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (122 ms) 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (64277 ms) 2024-04-29T20:25:55.767 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (65362 ms total) 2024-04-29T20:25:55.768 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: 2024-04-29T20:25:55.768 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-29T20:25:55.768 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (65362 ms total) 2024-04-29T20:25:55.768 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-29T20:25:56.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-29T20:25:56.007 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1490 ms) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (580 ms) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (60733 ms) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:24.398324+0000 osd.0 [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'DNEsmithi064-39012-1' : 1 ]) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:24.594079+0000 osd.1 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'AssertExistssmithi064-39395-1' : 1 ]) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:24.718896+0000 osd.6 [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'ceph_test_rados_open_pools_parallel.smithi064-38905' : 1 ]) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:25.432271+0000 osd.0 [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'DNEsmithi064-39012-1' : 1 ]) 2024-04-29T20:25:56.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:25.711881+0000 osd.1 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'AssertExistssmithi064-39395-1' : 1 ]) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:29.770316+0000 mon.b [INF] from='client.? 172.21.15.64:0/3978920568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38751-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:29.889372+0000 mon.b [INF] from='client.? 172.21.15.64:0/2781344944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38509-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:33.156591+0000 mon.a [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:37.517381+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38751-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:37.539513+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38509-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:33.647384+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:34.641298+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:35.683144+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:36.655616+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:37.714805+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:56.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:38.695503+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:57.051 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:56.969+0000 ea5d640 -1 osd.2 46 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4394.0:1 40.6s0 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-04-29T20:25:58.042 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:25:58.027+0000 ea5d640 -1 osd.2 46 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:25:58.385 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:39.749960+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:40.730904+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:41.698400+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:42.716955+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:43.326814+0000 mon.b [INF] from='client.? 172.21.15.64:0/2827380047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38434-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:43.968745+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:44.962046+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:46.178659+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:47.200200+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:47.299466+0000 mon.a [WRN] Health check update: 20 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:47.334592+0000 mon.a [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec, osd.2 has slow ops (SLOW_OPS) 2024-04-29T20:25:58.386 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:47.360693+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38434-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:48.176426+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:52.010365+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38751-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:52.012624+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38509-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:49.202962+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:50.232304+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:51.243951+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:52.245516+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:53.253637+0000 mon.b [INF] from='client.? 172.21.15.64:0/3770474016' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi064-38633-7"}]: dispatch 2024-04-29T20:25:58.387 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:53.268065+0000 mon.c [INF] from='client.? 172.21.15.64:0/3487481402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-29T20:26:00.096 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:00.013+0000 ea5d640 -1 osd.2 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:02.059 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:01.905+0000 ea5d640 -1 osd.2 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:03.171 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:03.132+0000 ea5d640 -1 osd.2 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:53.296450+0000 osd.2 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.777334+0000 mon.a [WRN] pool 'PoolQuotaPP_smithi064-38184-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.894134+0000 mon.a [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.899077+0000 mon.b [INF] from='client.? 172.21.15.64:0/1266692496' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi064-38267-10"}]: dispatch 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.903521+0000 mon.b [INF] from='client.? 172.21.15.64:0/209558667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi064-38288-10"}]: dispatch 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.991826+0000 mon.a [WRN] Health check failed: Slow OSD heartbeats on back (longest 8386.711ms) (OSD_SLOW_PING_TIME_BACK) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.992525+0000 mon.a [WRN] Health check failed: Slow OSD heartbeats on front (longest 8235.798ms) (OSD_SLOW_PING_TIME_FRONT) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.992977+0000 mon.a [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:54.993529+0000 mon.a [WRN] Health check update: 1 slow ops, oldest one blocked for 43 sec, osd.2 has slow ops (SLOW_OPS) 2024-04-29T20:26:03.357 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:55.075052+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi064-38633-7"}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:55.118552+0000 mon.b [INF] from='client.? 172.21.15.64:0/1760158833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi064-38204-16"}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:55.198557+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi064-38267-10"}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:55.235398+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi064-38288-10"}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:55.526704+0000 client.admin [INF] onexx 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:55.603810+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:25:56.125264+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi064-38204-16"}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.603841+0000 mon.b [INF] from='client.? 172.21.15.64:0/1055451813' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi064-38220-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.725648+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi064-38220-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-29T20:26:03.358 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.749361+0000 mon.c [INF] from='client.? 172.21.15.64:0/2183059613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi064-38757-1"}]: dispatch 2024-04-29T20:26:03.812 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.760775+0000 mon.c [INF] from='client.? 172.21.15.64:0/711375120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi064-38637-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.809727+0000 mon.c [INF] from='client.? 172.21.15.64:0/4123375347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi064-38538-7"}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.851623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi064-38757-1"}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.879399+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi064-38637-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.890006+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi064-38538-7"}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.914796+0000 mon.c [INF] from='client.? 172.21.15.64:0/4123375347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi064-38538-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.930953+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.360519+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi064-38538-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.485224+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi064-38170-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.515988+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi064-38434-4", "tierpool": "test-rados-api-smithi064-38434-4-cache"}]': finished 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.518441+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi064-38220-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.524697+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi064-38757-1"}]': finished 2024-04-29T20:26:03.813 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.529565+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi064-38637-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.910179+0000 mon.c [INF] from='client.? 172.21.15.64:0/711375120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi064-38637-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi064-38637-12"}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.918295+0000 mon.b [INF] from='client.? 172.21.15.64:0/2827380047' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi064-38434-4", "overlaypool": "test-rados-api-smithi064-38434-4-cache"}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.927848+0000 mon.b [INF] from='client.? 172.21.15.64:0/1055451813' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi064-38220-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi064-38220-23"}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.935718+0000 mon.b [INF] from='client.? 172.21.15.64:0/1053947763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38509-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.092871+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi064-38434-4", "overlaypool": "test-rados-api-smithi064-38434-4-cache"}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.104243+0000 mon.a [INF] from='client.4661 172.21.15.64:0/619696425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi064-38320-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.117744+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi064-38220-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi064-38220-23"}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.132012+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi064-38509-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.159846+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi064-38637-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi064-38637-12"}]: dispatch 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.742453+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi064-38633-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi064-38633-7"}]': finished 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.777903+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi064-38267-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi064-38267-10"}]': finished 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.781403+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi064-38204-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi064-38204-16"}]': finished 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.784811+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi064-38288-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi064-38288-10"}]': finished 2024-04-29T20:26:03.814 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.788213+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi064-38538-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-29T20:26:03.815 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.793011+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi064-38434-4", "overlaypool": "test-rados-api-smithi064-38434-4-cache"}]': finished 2024-04-29T20:26:03.815 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.801761+0000 mon.a [INF] from='client.4661 172.21.15.64:0/619696425' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi064-38320-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-29T20:26:04.494 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:02.806334+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"pr pool: Running main() from gmock_main.cc 2024-04-29T20:26:04.494 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-29T20:26:04.494 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [----------] Global test environment set-up. 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ OK ] NeoRadosPools.PoolList (6429 ms) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (6480 ms) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4952 ms) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (40409 ms) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (11775 ms) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3782 ms) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [----------] 6 tests from NeoRadosPools (73827 ms total) 2024-04-29T20:26:04.495 INFO:tasks.workunit.client.0.smithi064.stdout: pool: 2024-04-29T20:26:04.496 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [----------] Global test environment tear-down 2024-04-29T20:26:04.496 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [==========] 6 tests from 1 test suite ran. (73827 ms total) 2024-04-29T20:26:04.496 INFO:tasks.workunit.client.0.smithi064.stdout: pool: [ PASSED ] 6 tests. 2024-04-29T20:26:04.837 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:04.695+0000 ea5d640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout:efix": "osd pool application enable","pool": "test-rados-api-smithi064-38509-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.174140+0000 mon.b [INF] from='client.? 172.21.15.64:0/2827380047' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi064-38434-4-cache", "mode": "writeback"}]: dispatch 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.190115+0000 mon.c [INF] from='client.? 172.21.15.64:0/4123375347' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi064-38538-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi064-38538-7"}]: dispatch 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.217750+0000 mon.b [INF] from='client.? 172.21.15.64:0/342834507' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi064-38757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.246060+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi064-38434-4-cache", "mode": "writeback"}]: dispatch 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.284754+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi064-38757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.362942+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi064-38538-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi064-38538-7"}]: dispatch 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:00.015001+0000 osd.2 [WRN] 3 slow requests (by type [ 'started' : 2 'waiting for sub ops' : 1 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:26:05.054 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:01.906617+0000 osd.2 [WRN] 2 slow requests (by type [ 'started' : 2 ] most affected pool [ 'ListObjectssmithi064-39261-1' : 1 ]) 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.133287+0000 osd.2 [WRN] 1 slow requests (by type [ 'started' : 1 ] most affected pool [ 'ReadOpsmithi064-39190-2' : 1 ]) 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.346958+0000 mon.c [INF] from='client.? 172.21.15.64:0/3487481402' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.575290+0000 client.admin [INF] threexx 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.714060+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:03.969682+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:04.127781+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi064-38220-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi064-38220-23"}]': finished 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:04.130211+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi064-38637-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi064-38637-12"}]': finished 2024-04-29T20:26:05.055 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:04.132436+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi064-38434-4-cache", "mode": "writeback"}]': finished 2024-04-29T20:26:05.291 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:04:23.797 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.307 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:04:23.812 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.318 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:04:23.824 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.365 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:23.841 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.371 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:04:23.877 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.385 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:23.862 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.479 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:04:23.985 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.527 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:24.004 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.538 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:04:24.044 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:26:05.909 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:05.905+0000 ea5d640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:06.960 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:06.955+0000 ea5d640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:08.562 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: got: 2024-04-29T20:26:04.134314+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi064-38757-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-29T20:26:08.563 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (15394 ms) 2024-04-29T20:26:08.563 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (78198 ms total) 2024-04-29T20:26:08.563 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: 2024-04-29T20:26:08.563 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-29T20:26:08.563 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (78198 ms total) 2024-04-29T20:26:08.563 INFO:tasks.workunit.client.0.smithi064.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-29T20:26:12.433 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:26:12.285+0000 ea5d640 -1 osd.2 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4623.0:1 52.0s0 52:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2024-04-29T20:26:28.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: Running main() from gmock_main.cc 2024-04-29T20:26:28.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [==========] Running 38 tests from 2 test suites. 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: seed 38220 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (34030 ms) 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (184 ms) 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-29T20:26:28.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (277 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (20061 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (61 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (82 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (103 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (52 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (81 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (363 ms) 2024-04-29T20:26:28.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (79 ms) 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (210 ms) 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (62 ms) 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (79 ms) 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (104 ms) 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-29T20:26:28.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (406 ms) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (94 ms) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (70 ms) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (50 ms) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (70 ms) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP (56519 ms total) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (22991 ms) 2024-04-29T20:26:28.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-29T20:26:28.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (612 ms) 2024-04-29T20:26:28.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-29T20:26:28.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (231 ms) 2024-04-29T20:26:28.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.R list_parallel: process_1_[39024]: starting. 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_1_[39024]: creating pool ceph_test_rados_list_parallel.smithi064-38942 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_1_[39024]: created object 0... 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_1_[39024]: created object 25... 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_1_[39024]: created object 49... 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_1_[39024]: finishing. 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_1_[39024]: shutting down. 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_2_[39026]: starting. 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_2_[39026]: listing objects. 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_2_[39026]: listed object 0... 2024-04-29T20:26:33.393 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_2_[39026]: listed object 25... 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_2_[39026]: saw 50 objects 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_2_[39026]: shutting down. 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: starting. 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: creating pool ceph_test_rados_list_parallel.smithi064-38942 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: created object 0... 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: created object 25... 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: created object 49... 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: finishing. 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_3_[40408]: shutting down. 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_4_[40409]: starting. 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_4_[40409]: listing objects. 2024-04-29T20:26:33.394 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_4_[40409]: listed object 0... 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_4_[40409]: saw 10 objects 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_4_[40409]: shutting down. 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_5_[40410]: starting. 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_5_[40410]: removed 25 objects... 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_5_[40410]: removed half of the objects 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_5_[40410]: removed 50 objects... 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_5_[40410]: removed 50 objects 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_5_[40410]: shutting down. 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: starting. 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: creating pool ceph_test_rados_list_parallel.smithi064-38942 2024-04-29T20:26:33.395 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: created object 0... 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: created object 25... 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: created object 49... 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: finishing. 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_6_[40472]: shutting down. 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: starting. 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: listing objects. 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: listed object 0... 2024-04-29T20:26:33.396 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: listed object 25... 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: listed object 50... 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: saw 52 objects 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_7_[40473]: shutting down. 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_8_[40474]: starting. 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_8_[40474]: added 25 objects... 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_8_[40474]: added half of the objects 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_8_[40474]: added 50 objects... 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_8_[40474]: added 50 objects 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_8_[40474]: shutting down. 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.397 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.398 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:26:33.398 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_9_[40969]: starting. 2024-04-29T20:26:37.058 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-29T20:26:37.058 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-29T20:26:37.058 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-29T20:26:37.058 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: handle_notify cookie 94762409694064 notify_id 176093659136 notifier_gid 4403 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (58242 ms) 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: Trying... 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: handle_notify cookie 94762423227152 notify_id 270582939649 notifier_gid 4786 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: Waiting for 3.000000000s 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: Timed out. 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: Flushing... 2024-04-29T20:26:37.059 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: Flushed... 2024-04-29T20:26:37.060 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (48114 ms) 2024-04-29T20:26:37.060 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (106356 ms total) 2024-04-29T20:26:37.060 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: 2024-04-29T20:26:37.060 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-29T20:26:37.060 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (106356 ms total) 2024-04-29T20:26:37.060 INFO:tasks.workunit.client.0.smithi064.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-29T20:26:37.068 INFO:tasks.workunit.client.0.smithi064.stdout: cls: Running main() from gmock_main.cc 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [----------] Global test environment set-up. 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [ OK ] NeoRadosCls.DNE (56855 ms) 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (49662 ms) 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [----------] 2 tests from NeoRadosCls (106517 ms total) 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [----------] Global test environment tear-down 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [==========] 2 tests from 1 test suite ran. (106518 ms total) 2024-04-29T20:26:37.069 INFO:tasks.workunit.client.0.smithi064.stdout: cls: [ PASSED ] 2 tests. 2024-04-29T20:26:37.162 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: starting. 2024-04-29T20:26:37.162 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: creating pool ceph_test_rados_open_pools_parallel.smithi064-38905 2024-04-29T20:26:37.162 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: created object 0... 2024-04-29T20:26:37.162 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: created object 25... 2024-04-29T20:26:37.162 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: created object 49... 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: finishing. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_1_[38928]: shutting down. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_2_[38929]: starting. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_2_[38929]: rados_pool_create. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_2_[38929]: rados_ioctx_create. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_2_[38929]: shutting down. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: ******************************* 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: starting. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: creating pool ceph_test_rados_open_pools_parallel.smithi064-38905 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: created object 0... 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: created object 25... 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: created object 49... 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: finishing. 2024-04-29T20:26:37.163 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_3_[40641]: shutting down. 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: ******************************* 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_4_[40642]: starting. 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_4_[40642]: rados_pool_create. 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_4_[40642]: rados_ioctx_create. 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: process_4_[40642]: shutting down. 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: ******************************* 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: ******************************* 2024-04-29T20:26:37.164 INFO:tasks.workunit.client.0.smithi064.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: Running main() from gmock_main.cc 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [----------] Global test environment set-up. 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6531 ms) 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (17992 ms) 2024-04-29T20:26:37.171 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-29T20:26:37.172 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (82071 ms) 2024-04-29T20:26:37.172 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [----------] 3 tests from NeoRadosCmd (106594 ms total) 2024-04-29T20:26:37.172 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: 2024-04-29T20:26:37.172 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [----------] Global test environment tear-down 2024-04-29T20:26:37.172 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [==========] 3 tests from 1 test suite ran. (106594 ms total) 2024-04-29T20:26:37.172 INFO:tasks.workunit.client.0.smithi064.stdout: cmd: [ PASSED ] 3 tests. 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: Running main() from gmock_main.cc 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [----------] Global test environment set-up. 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (10895 ms) 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (97895 ms) 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (108790 ms total) 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [----------] Global test environment tear-down 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (108790 ms total) 2024-04-29T20:26:39.411 INFO:tasks.workunit.client.0.smithi064.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: Running main() from gmock_main.cc 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [----------] Global test environment set-up. 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (18730 ms) 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.LockShared (6531 ms) 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-29T20:26:39.700 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1389 ms) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1627 ms) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (6380 ms) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.Unlock (138 ms) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (177 ms) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (76 ms) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [----------] 8 tests from LibRadosLock (35048 ms total) 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-29T20:26:39.701 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (16913 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6100 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1368 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1217 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (2280 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (216 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (302 ms) 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-29T20:26:39.702 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (242 ms) 2024-04-29T20:26:39.703 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (28638 ms total) 2024-04-29T20:26:39.703 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: 2024-04-29T20:26:39.703 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [----------] Global test environment tear-down 2024-04-29T20:26:39.703 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (109520 ms total) 2024-04-29T20:26:39.703 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout:oundTripPP (260 ms) 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (336 ms) 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (1896 ms) 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-29T20:26:39.726 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (287 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (129 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (240 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (145 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (232 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (179 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (542 ms) 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-29T20:26:39.727 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (227 ms) 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (253 ms) 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (154 ms) 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (253 ms) 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (28968 ms total) 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [==========] 38 tests from 2 test suites ran. (109568 ms total) 2024-04-29T20:26:39.728 INFO:tasks.workunit.client.0.smithi064.stdout: api_io_pp: [ PASSED ] 38 tests. 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify_test_cb 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (23173 ms) 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-29T20:26:39.798 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94055640959200 err -107 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (3748 ms) 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94055640959200 err -107 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (3017 ms) 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561024 cookie 94055641023104 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (4846 ms) 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561025 cookie 94055641075584 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (169 ms) 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-29T20:26:39.799 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561024 cookie 94055641055328 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (647 ms) 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561026 cookie 94055641055328 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561026 cookie 94055641101456 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (247 ms) 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561025 cookie 94055641103584 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561026 cookie 94055641103584 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3214 ms) 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94055641060368 err -107 2024-04-29T20:26:39.800 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_cb from 4340 notify_id 188978561028 cookie 94055641103584 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5786 ms) 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94055641103584 err -107 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2934 ms) 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (47781 ms total) 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: watch_notify_test_cb 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (21933 ms) 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (21933 ms total) 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-29T20:26:39.801 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (109442 ms total) 2024-04-29T20:26:39.802 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: Running main() from gmock_main.cc 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [----------] Global test environment set-up. 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolList (6665 ms) 2024-04-29T20:26:56.103 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (8048 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (11902 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (42135 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5452 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (9362 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (23199 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6062 ms) 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-29T20:26:56.104 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (12982 ms) 2024-04-29T20:26:56.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [----------] 9 tests from LibRadosPools (125807 ms total) 2024-04-29T20:26:56.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: 2024-04-29T20:26:56.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [----------] Global test environment tear-down 2024-04-29T20:26:56.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (125807 ms total) 2024-04-29T20:26:56.105 INFO:tasks.workunit.client.0.smithi064.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-29T20:27:06.026 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-04-29T20:27:06.026 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: seed 38633 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (27211 ms) 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (55 ms) 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (2156 ms) 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-29T20:27:06.027 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3053 ms) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (2621 ms) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (35097 ms total) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (25060 ms) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2870 ms) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (6551 ms) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (22346 ms) 2024-04-29T20:27:06.028 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (56827 ms total) 2024-04-29T20:27:06.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: 2024-04-29T20:27:06.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-29T20:27:06.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (135671 ms total) 2024-04-29T20:27:06.029 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-29T20:27:06.897 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: Running main() from gmock_main.cc 2024-04-29T20:27:06.897 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-29T20:27:06.897 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [----------] Global test environment set-up. 2024-04-29T20:27:06.897 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-29T20:27:06.897 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStat.Stat (33862 ms) 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (140 ms) 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStat.StatNS (443 ms) 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (8027 ms) 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (11933 ms) 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [----------] 5 tests from LibRadosStat (54405 ms total) 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-29T20:27:06.898 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (22527 ms) 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (29081 ms) 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (174 ms) 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2146 ms) 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (53928 ms total) 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [----------] Global test environment tear-down 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (136581 ms total) 2024-04-29T20:27:06.899 INFO:tasks.workunit.client.0.smithi064.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-29T20:27:18.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-29T20:27:18.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-29T20:27:18.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-29T20:27:18.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-29T20:27:18.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (30038 ms) 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (17521 ms) 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (47559 ms total) 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (20212 ms) 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-29T20:27:18.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (35511 ms) 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (14375 ms) 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (400 ms) 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418579296 notify_id 377957122048 notifier_gid 4879 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (758 ms) 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886419229024 notify_id 377957122049 notifier_gid 4879 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (380 ms) 2024-04-29T20:27:18.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886419229024 notify_id 377957122050 notifier_gid 4879 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (405 ms) 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418587888 notify_id 377957122050 notifier_gid 4879 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (350 ms) 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418587888 notify_id 377957122051 notifier_gid 4879 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (305 ms) 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418587888 notify_id 377957122049 notifier_gid 4879 2024-04-29T20:27:18.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (165 ms) 2024-04-29T20:27:18.736 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-29T20:27:18.736 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: trying... 2024-04-29T20:27:18.736 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418587888 notify_id 377957122050 notifier_gid 4879 2024-04-29T20:27:18.736 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: timed out 2024-04-29T20:27:27.587 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: flu list_parallel: process_9_[40969]: creating pool ceph_test_rados_list_parallel.smithi064-38942 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_9_[40969]: created object 0... 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_9_[40969]: created object 25... 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_9_[40969]: created object 49... 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_9_[40969]: finishing. 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_9_[40969]: shutting down. 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: starting. 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: listing objects. 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: listed object 0... 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: listed object 25... 2024-04-29T20:27:27.588 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: listed object 50... 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: listed object 75... 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: listed object 100... 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: saw 103 objects 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_10_[40970]: shutting down. 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_13_[40973]: starting. 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_13_[40973]: removed 25 objects... 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_13_[40973]: removed half of the objects 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_13_[40973]: removed 50 objects... 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_13_[40973]: removed 50 objects 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_13_[40973]: shutting down. 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.589 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_11_[40971]: starting. 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_11_[40971]: added 25 objects... 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_11_[40971]: added half of the objects 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_11_[40971]: added 50 objects... 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_11_[40971]: added 50 objects 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_11_[40971]: shutting down. 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_12_[40972]: starting. 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_12_[40972]: added 25 objects... 2024-04-29T20:27:27.590 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_12_[40972]: added half of the objects 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_12_[40972]: added 50 objects... 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_12_[40972]: added 50 objects 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_12_[40972]: shutting down. 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: starting. 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: creating pool ceph_test_rados_list_parallel.smithi064-38942 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: created object 0... 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: created object 25... 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: created object 49... 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: finishing. 2024-04-29T20:27:27.591 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_14_[48261]: shutting down. 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: starting. 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listing objects. 2024-04-29T20:27:27.592 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listed object 0... 2024-04-29T20:27:34.492 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:52.997 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:34.500 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:53.006 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.046 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listed objec delete_pools_parallel: process_1_[38953]: starting. 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_1_[38953]: creating pool ceph_test_rados_delete_pools_parallel.smithi064-38917 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_1_[38953]: created object 0... 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_1_[38953]: created object 25... 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_1_[38953]: created object 49... 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_1_[38953]: finishing. 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_1_[38953]: shutting down. 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_2_[38954]: starting. 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_2_[38954]: deleting pool ceph_test_rados_delete_pools_parallel.smithi064-38917 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_2_[38954]: shutting down. 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: ******************************* 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: starting. 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: creating pool ceph_test_rados_delete_pools_parallel.smithi064-38917 2024-04-29T20:27:35.047 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: created object 0... 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: created object 25... 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: created object 49... 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: finishing. 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_3_[40705]: shutting down. 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: ******************************* 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_5_[40707]: starting. 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_5_[40707]: listing objects. 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_5_[40707]: listed object 0... 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_5_[40707]: listed object 25... 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_5_[40707]: saw 50 objects 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_5_[40707]: shutting down. 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: ******************************* 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_4_[40706]: starting. 2024-04-29T20:27:35.048 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_4_[40706]: deleting pool ceph_test_rados_delete_pools_parallel.smithi064-38917 2024-04-29T20:27:35.049 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: process_4_[40706]: shutting down. 2024-04-29T20:27:35.049 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: ******************************* 2024-04-29T20:27:35.049 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: ******************************* 2024-04-29T20:27:35.049 INFO:tasks.workunit.client.0.smithi064.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-29T20:27:35.233 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:53.709 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.241 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:53.718 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.442 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:53.948 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.465 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:53.971 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.574 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:54.051 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.582 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:54.088 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:35.865 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:54.371 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:27:43.032 INFO:tasks.workunit.client.0.smithi064.stdout:t 25... 2024-04-29T20:27:43.032 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listed object 50... 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listed object 75... 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listed object 100... 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: listed object 125... 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: saw 150 objects 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_15_[48265]: shutting down. 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_16_[48266]: starting. 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_16_[48266]: added 25 objects... 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_16_[48266]: added half of the objects 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_16_[48266]: added 50 objects... 2024-04-29T20:27:43.033 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_16_[48266]: added 50 objects 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: process_16_[48266]: shutting down. 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******************************* 2024-04-29T20:27:43.034 INFO:tasks.workunit.client.0.smithi064.stdout: list_parallel: ******* SUCCESS ********** 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: Running main() from gmock_main.cc 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [----------] Global test environment set-up. 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (14286 ms) 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: no timeout :/ 2024-04-29T20:27:52.719 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: no timeout :/ 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: no timeout :/ 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: no timeout :/ 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: no timeout :/ 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (15565 ms) 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (615 ms) 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.Checksum (1009 ms) 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (2723 ms) 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (281 ms) 2024-04-29T20:27:52.720 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (941 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (61 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.TruncTest (108 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (213 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (80 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.RmXattr (2093 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIo.XattrIter (458 ms) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [----------] 14 tests from LibRadosIo (38433 ms total) 2024-04-29T20:27:52.721 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (52299 ms) 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (3746 ms) 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (734 ms) 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4180 ms) 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7445 ms) 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (2936 ms) 2024-04-29T20:27:52.722 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (919 ms) 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (183 ms) 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (901 ms) 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (267 ms) 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [----------] 10 tests from LibRadosIoEC (73610 ms total) 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: 2024-04-29T20:27:52.723 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [----------] Global test environment tear-down 2024-04-29T20:27:52.724 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [==========] 24 tests from 2 test suites ran. (182553 ms total) 2024-04-29T20:27:52.724 INFO:tasks.workunit.client.0.smithi064.stdout: api_io: [ PASSED ] 24 tests. 2024-04-29T20:27:52.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-29T20:27:52.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-29T20:27:52.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-29T20:27:52.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-29T20:27:52.732 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: seed 38288 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (30317 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (198 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1160 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1778 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (1239 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (953 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (135 ms) 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-29T20:27:52.733 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (121 ms) 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (35901 ms total) 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (43192 ms) 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (9728 ms) 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (6781 ms) 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (8744 ms) 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-29T20:27:52.734 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3547 ms) 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (697 ms) 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (575 ms) 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (219 ms) 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (73483 ms total) 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (182519 ms total) 2024-04-29T20:27:52.735 INFO:tasks.workunit.client.0.smithi064.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: Running main() from gmock_main.cc 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [----------] Global test environment set-up. 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [----------] 3 tests from NeoradosList 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ OK ] NeoradosList.ListObjects (65135 ms) 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ OK ] NeoradosList.ListObjectsNS (41379 ms) 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ OK ] NeoradosList.ListObjectsMany (88984 ms) 2024-04-29T20:28:06.139 INFO:tasks.workunit.client.0.smithi064.stdout: list: [----------] 3 tests from NeoradosList (195498 ms total) 2024-04-29T20:28:06.140 INFO:tasks.workunit.client.0.smithi064.stdout: list: 2024-04-29T20:28:06.140 INFO:tasks.workunit.client.0.smithi064.stdout: list: [----------] Global test environment tear-down 2024-04-29T20:28:06.140 INFO:tasks.workunit.client.0.smithi064.stdout: list: [==========] 3 tests from 1 test suite ran. (195498 ms total) 2024-04-29T20:28:06.140 INFO:tasks.workunit.client.0.smithi064.stdout: list: [ PASSED ] 3 tests. 2024-04-29T20:28:20.212 INFO:tasks.workunit.client.0.smithi064.stdout:shing 2024-04-29T20:28:20.212 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: flushed 2024-04-29T20:28:20.212 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3308 ms) 2024-04-29T20:28:20.212 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-29T20:28:20.212 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: trying... 2024-04-29T20:28:20.212 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418587888 notify_id 382252089347 notifier_gid 4879 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: timed out 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: flushing 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: flushed 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3676 ms) 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: List watches 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify2 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418548384 notify_id 386547056642 notifier_gid 4879 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify2 done 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: watch_check 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: unwatch2 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: flushing 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: done 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (5253 ms) 2024-04-29T20:28:20.213 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: List watches 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify2 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: handle_notify cookie 94886418587888 notify_id 386547056644 notifier_gid 4879 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: notify2 done 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: watch_check 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: unwatch2 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: flushing 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: done 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3358 ms) 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (88457 ms total) 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (209794 ms total) 2024-04-29T20:28:20.214 INFO:tasks.workunit.client.0.smithi064.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-29T20:28:22.392 INFO:tasks.workunit.client.0.smithi064.stdout:575feb40320 2024-04-29T20:28:22.392 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb40e20 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb3f550 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb41cd0 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb40b90 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb42c00 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb41b10 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb43b70 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb42910 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb446b0 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb45140 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb43870 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb45c50 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb466e0 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb44ef0 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb475b0 2024-04-29T20:28:22.393 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb46460 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb48110 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb48bc0 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb47320 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb49720 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb4a1e0 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb48940 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: waiting 0x5575feb4ad20 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: done. 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (66519 ms) 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (66519 ms total) 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-29T20:28:22.394 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.Exec (52146 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (2992 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (1614 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.Applications (7991 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-29T20:28:22.395 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-29T20:28:22.396 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (73334 ms) 2024-04-29T20:28:22.396 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] 8 tests from LibRadosMisc (138077 ms total) 2024-04-29T20:28:22.396 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: 2024-04-29T20:28:22.396 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [----------] Global test environment tear-down 2024-04-29T20:28:22.396 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (212147 ms total) 2024-04-29T20:28:22.396 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-29T20:28:27.662 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:46.168 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:27.684 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:46.189 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:27.697 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:46.203 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:27.720 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:46.196 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:27.722 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:46.199 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:27.726 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:46.232 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:28.015 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:46.492 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:28.021 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:46.526 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:28.167 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:46.673 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:52.845 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:11.350 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:52.850 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:11.356 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:52.924 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:11.400 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:52.925 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:11.402 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:52.946 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:11.451 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:52.956 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:11.461 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:53.010 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:11.516 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:53.026 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:11.502 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:28:53.080 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:11.586 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:07.425 INFO:tasks.workunit.client.0.smithi064.stdout: misc: Running main() from gmock_main.cc 2024-04-29T20:29:07.425 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [----------] Global test environment set-up. 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.Version (4671 ms) 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (5367 ms) 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.LongName (56688 ms) 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (43536 ms) 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-29T20:29:07.426 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (43526 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (41698 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.Exec (27241 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (8621 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (4930 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.BigObject (13604 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2200 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (4680 ms) 2024-04-29T20:29:07.427 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [----------] 12 tests from NeoRadosMisc (256762 ms total) 2024-04-29T20:29:07.428 INFO:tasks.workunit.client.0.smithi064.stdout: misc: 2024-04-29T20:29:07.428 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [----------] Global test environment tear-down 2024-04-29T20:29:07.428 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [==========] 12 tests from 1 test suite ran. (256762 ms total) 2024-04-29T20:29:07.428 INFO:tasks.workunit.client.0.smithi064.stdout: misc: [ PASSED ] 12 tests. 2024-04-29T20:29:13.079 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:31.585 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.087 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:31.593 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.107 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:31.584 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.108 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:31.614 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.117 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:31.622 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.218 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:31.695 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.313 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:31.819 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.339 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:31.815 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:13.348 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:31.854 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.099 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:51.604 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.231 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:51.737 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.322 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:51.828 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.381 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:51.857 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.389 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:51.866 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.459 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:51.935 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.463 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:51.969 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.468 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:51.974 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:33.559 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:52.065 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.476 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:11.982 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.479 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:11.985 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.521 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:12.027 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.554 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:12.031 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.558 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:12.035 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.565 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:12.068 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.577 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:12.083 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.634 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:12.111 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:53.657 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:12.162 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:29:57.873 INFO:tasks.workunit.client.0.smithi064.stdout: io: Running main() from gmock_main.cc 2024-04-29T20:29:57.873 INFO:tasks.workunit.client.0.smithi064.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-29T20:29:57.873 INFO:tasks.workunit.client.0.smithi064.stdout: io: [----------] Global test environment set-up. 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.Limits (45857 ms) 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (57464 ms) 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.ReadOp (28813 ms) 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.SparseRead (21928 ms) 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.RoundTrip (41431 ms) 2024-04-29T20:29:57.874 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (28278 ms) 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (5432 ms) 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (26547 ms) 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (6446 ms) 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.Trunc (9416 ms) 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.Remove (10088 ms) 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-29T20:29:57.875 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (10536 ms) 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.RmXattr (8964 ms) 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ OK ] NeoRadosIo.GetXattrs (6043 ms) 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [----------] 14 tests from NeoRadosIo (307244 ms total) 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [----------] Global test environment tear-down 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [==========] 14 tests from 1 test suite ran. (307244 ms total) 2024-04-29T20:29:57.876 INFO:tasks.workunit.client.0.smithi064.stdout: io: [ PASSED ] 14 tests. 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: Running main() from gmock_main.cc 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [----------] Global test environment set-up. 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (42624 ms) 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (61574 ms) 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-29T20:30:01.671 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (27879 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (22016 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (29532 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (37630 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (14047 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (5944 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (9722 ms) 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-29T20:30:01.672 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (8014 ms) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (7083 ms) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (25028 ms) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (8885 ms) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (11016 ms) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (310995 ms total) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [----------] Global test environment tear-down 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (310995 ms total) 2024-04-29T20:30:01.673 INFO:tasks.workunit.client.0.smithi064.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-29T20:30:03.854 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: Running main() from gmock_main.cc 2024-04-29T20:30:03.854 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-29T20:30:03.854 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-29T20:30:03.854 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (59317 ms) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (40460 ms) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (25538 ms) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5113 ms) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (130428 ms total) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (42708 ms) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (12336 ms) 2024-04-29T20:30:03.855 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (14326 ms) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (69370 ms total) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (6560 ms) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (3592 ms) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (5515 ms) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (3161 ms) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (18828 ms total) 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-29T20:30:03.856 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (27870 ms) 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (13460 ms) 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (41330 ms total) 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (313512 ms total) 2024-04-29T20:30:03.857 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-29T20:30:11.528 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-04-29T20:30:11.528 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-29T20:30:11.528 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-29T20:30:11.528 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: seed 38464 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (59300 ms) 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (6496 ms) 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (58411 ms) 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (4148 ms) 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (37459 ms) 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (165814 ms total) 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: 2024-04-29T20:30:11.529 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (18623 ms) 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (14079 ms) 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (22678 ms) 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (9008 ms) 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2593 ms) 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3150-g354447ca/rpm/el9/BUILD/ceph-19.0.0-3150-g354447ca/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: 2024-04-29T20:30:11.530 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi064-38464-7 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (20807 ms) 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (87788 ms total) 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (6177 ms) 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (6184 ms total) 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-29T20:30:11.531 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-29T20:30:11.532 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (6710 ms) 2024-04-29T20:30:11.532 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-29T20:30:13.729 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:32.234 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.737 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:32.243 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.807 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:32.283 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.810 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:32.316 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.812 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:32.289 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.828 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:32.334 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.841 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:32.347 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.883 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:32.360 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:13.907 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:32.413 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:25.030 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] Li snapshots: Running main() from gmock_main.cc 2024-04-29T20:30:25.030 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-29T20:30:25.030 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [----------] Global test environment set-up. 2024-04-29T20:30:25.030 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (69899 ms) 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (37792 ms) 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (46527 ms) 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (58724 ms) 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (34603 ms) 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (247545 ms total) 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: 2024-04-29T20:30:25.031 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (12487 ms) 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (11509 ms) 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (22090 ms) 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (10220 ms) 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (8218 ms) 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi064-39353-11. 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: Waiting for snaps to purge. 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (22266 ms) 2024-04-29T20:30:25.032 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (86790 ms total) 2024-04-29T20:30:25.033 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: 2024-04-29T20:30:25.033 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [----------] Global test environment tear-down 2024-04-29T20:30:25.033 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (334335 ms total) 2024-04-29T20:30:25.033 INFO:tasks.workunit.client.0.smithi064.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: seed 38357 2024-04-29T20:30:29.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (565 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (30644 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (4805 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (392 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (329 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (77 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (61 ms) 2024-04-29T20:30:29.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (163 ms) 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (45 ms) 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (355 ms) 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (108 ms) 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (85 ms) 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (265 ms) 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-29T20:30:29.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (160607 ms) 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (10147 ms) 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: done waiting, doing copies 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: done waiting 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (116314 ms) 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (1048 ms) 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (812 ms) 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4393 ms) 2024-04-29T20:30:29.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-29T20:30:29.926 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-29T20:30:29.926 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-29T20:30:29.926 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-29T20:30:29.926 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-29T20:30:29.926 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-29T20:30:33.545 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:52.050 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.548 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:52.054 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.621 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:52.127 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.623 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:52.129 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.627 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:52.133 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.629 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:52.105 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.633 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:52.110 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.702 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:52.179 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:33.714 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:52.220 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] api_aio: Running main() from gmock_main.cc 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [----------] Global test environment set-up. 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6881 ms) 2024-04-29T20:30:41.921 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (55402 ms) 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (42470 ms) 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (21199 ms) 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (28811 ms) 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (41292 ms) 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (26203 ms) 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-29T20:30:41.922 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (8542 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (8824 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (17596 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (10496 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4378 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (11504 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (8041 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.Flush (6032 ms) 2024-04-29T20:30:41.923 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (7831 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4921 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4278 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3554 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4384 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (5727 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4317 ms) 2024-04-29T20:30:41.924 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4194 ms) 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (5008 ms) 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4639 ms) 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (5256 ms) 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [----------] 26 tests from LibRadosAio (351781 ms total) 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: 2024-04-29T20:30:41.925 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-29T20:30:43.176 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.Sim api_aio_pp: Running main() from gmock_main.cc 2024-04-29T20:30:43.176 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-29T20:30:43.176 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6942 ms) 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (77765 ms) 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (48014 ms) 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (22178 ms) 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (29287 ms) 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (30266 ms) 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-29T20:30:43.177 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (9804 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (6487 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (6782 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (5098 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (6510 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (8090 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (7256 ms) 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-29T20:30:43.178 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (11164 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (6549 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (10510 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (12788 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4907 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (4279 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3561 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3150 ms) 2024-04-29T20:30:43.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (5159 ms) 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4011 ms) 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (6301 ms) 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (6071 ms) 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (6649 ms) 2024-04-29T20:30:43.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRbRadosSnapshotsECPP.SnapRemovePP (3148 ms) 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (6975 ms) 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3272 ms) 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (20105 ms total) 2024-04-29T20:30:52.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (8678 ms) 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6918 ms) 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5188 ms) 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (20784 ms total) 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (361881 ms total) 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-29T20:30:52.151 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-29T20:30:52.152 INFO:tasks.workunit.client.0.smithi064.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-29T20:30:53.861 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:12.367 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.862 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:12.339 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.865 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:12.342 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.870 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:12.376 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.936 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:12.412 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.957 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:12.462 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.961 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:12.466 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:53.963 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:12.469 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:30:54.048 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:12.553 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.161 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:20.666 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.245 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:20.750 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.252 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:20.758 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.291 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:20.797 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.320 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:20.797 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.333 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:20.809 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.343 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:20.849 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.392 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:20.869 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:02.411 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:20.917 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:07.188 INFO:tasks.workunit.client.0.smithi064.stdout: 22 tests from LibRadosMiscPP (331215 ms total) 2024-04-29T20:31:07.188 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:31:07.188 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (7150 ms) 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (7150 ms total) 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (1173 ms) 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (34 ms) 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (1207 ms total) 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-29T20:31:07.189 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (948 ms) 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (620 ms) 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (1568 ms total) 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (602 ms) 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (45 ms) 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (647 ms total) 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-29T20:31:07.190 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1858 ms) 2024-04-29T20:31:07.191 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1858 ms total) 2024-04-29T20:31:07.191 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: 2024-04-29T20:31:07.191 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-29T20:31:07.191 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (376937 ms total) 2024-04-29T20:31:07.191 INFO:tasks.workunit.client.0.smithi064.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-29T20:31:14.008 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:32.513 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.016 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:32.522 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.076 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:32.582 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.085 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:32.561 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.088 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:32.564 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.094 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:32.599 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.100 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:32.606 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.131 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:32.608 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:14.150 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:32.656 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.314 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:57.819 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.317 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:57.823 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.368 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:57.844 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.370 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:57.847 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.383 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:57.889 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.386 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:57.892 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.398 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:57.904 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.441 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:57.918 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:39.487 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:57.992 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=7 expected=7 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=2 expected=2 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:e960b815:::9:head 2024-04-29T20:31:53.145 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=9 expected=9 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=12 expected=12 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=4 expected=4 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-04-29T20:31:53.146 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=1 expected=1 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=10 expected=10 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : seek to 9:566253c9:::13:head 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: : entry=13 expected=13 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (9676 ms) 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (322826 ms) 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2024-04-29T20:31:53.147 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (32271 ms) 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] 7 tests from LibRadosList (406322 ms total) 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2227 ms) 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset foo1,foo2,foo3 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo1 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo2 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo3 2024-04-29T20:31:53.148 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset foo1,foo4,foo5 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo4 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo5 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo1 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset foo6,foo7 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo7 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: foo6 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns1:foo4 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns1:foo5 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns2:foo7 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns2:foo6 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: ns1:foo1 2024-04-29T20:31:53.149 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: :foo1 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: :foo2 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: :foo3 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (2081 ms) 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 1 0 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 10 0 2024-04-29T20:31:53.150 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 13 0 2024-04-29T20:31:53.768 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: Running main() from gmock_main.cc 2024-04-29T20:31:53.768 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-29T20:31:53.768 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [----------] Global test environment set-up. 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (109116 ms) 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (66515 ms) 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (247494 ms) 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [----------] 3 tests from NeoradosECList (423125 ms total) 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [----------] Global test environment tear-down 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (423126 ms total) 2024-04-29T20:31:53.769 INFO:tasks.workunit.client.0.smithi064.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-29T20:31:59.316 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:17.821 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.322 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:17.828 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.370 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:17.846 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.373 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:17.850 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.375 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:17.881 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.385 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:17.891 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.391 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:17.897 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.426 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:17.902 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:31:59.451 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:17.956 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:00.905 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 7 0 2024-04-29T20:32:00.905 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 14 0 2024-04-29T20:32:00.905 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 0 0 2024-04-29T20:32:00.905 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 15 0 2024-04-29T20:32:00.905 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 11 0 2024-04-29T20:32:00.905 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 5 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 8 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 6 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 3 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 4 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 12 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 9 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 2 0 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3575 ms) 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] 3 tests from LibRadosListEC (7883 ms total) 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-29T20:32:00.906 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-29T20:32:00.907 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3338 ms) 2024-04-29T20:32:00.907 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] 1 test from LibRadosListNP (3338 ms total) 2024-04-29T20:32:00.907 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: 2024-04-29T20:32:00.907 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [----------] Global test environment tear-down 2024-04-29T20:32:00.907 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [==========] 11 tests from 3 test suites ran. (430718 ms total) 2024-04-29T20:32:00.907 INFO:tasks.workunit.client.0.smithi064.stdout: api_list: [ PASSED ] 11 tests. 2024-04-29T20:32:08.715 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: Running main() from gmock_main.cc 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [----------] Global test environment set-up. 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (64070 ms) 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (34952 ms) 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (19932 ms) 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (14629 ms) 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-29T20:32:08.716 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (38901 ms) 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (36545 ms) 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (228984 ms) 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (438013 ms total) 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [----------] Global test environment tear-down 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (438013 ms total) 2024-04-29T20:32:08.717 INFO:tasks.workunit.client.0.smithi064.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-29T20:32:11.872 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-04-29T20:32:11.872 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-29T20:32:11.872 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-29T20:32:11.872 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-29T20:32:11.872 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-29T20:32:11.872 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (27671 ms) 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (57407 ms) 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (21664 ms) 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-29T20:32:11.873 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (25788 ms) 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (21961 ms) 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (37824 ms) 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (249141 ms) 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (441456 ms total) 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (441456 ms total) 2024-04-29T20:32:11.874 INFO:tasks.workunit.client.0.smithi064.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-29T20:32:19.415 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:37.920 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.418 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:37.924 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.473 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:37.979 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.474 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:37.950 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.476 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:37.953 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.484 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:37.990 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.487 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:37.993 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.520 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:37.997 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:19.545 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:38.051 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.721 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:58.227 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.774 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:58.251 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.775 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:58.252 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.789 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:58.294 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.793 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:58.299 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.804 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:58.310 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.840 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:58.317 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.864 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:58.370 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:39.864 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:58.370 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:49.684 INFO:tasks.workunit.client.0.smithi064.stdout:pleWrite 2024-04-29T20:32:49.684 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8029 ms) 2024-04-29T20:32:49.684 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-29T20:32:49.684 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8987 ms) 2024-04-29T20:32:49.684 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-29T20:32:49.684 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (8423 ms) 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7813 ms) 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (9231 ms) 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (9092 ms) 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (9186 ms) 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6805 ms) 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-29T20:32:49.685 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8163 ms) 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6749 ms) 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (8483 ms) 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7090 ms) 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (8153 ms) 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6995 ms) 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-29T20:32:49.686 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7649 ms) 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6911 ms) 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (127759 ms total) 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [----------] Global test environment tear-down 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (479542 ms total) 2024-04-29T20:32:49.687 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-29T20:32:59.185 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:17.691 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.199 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:17.705 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.207 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:17.684 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.212 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:17.718 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.214 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:17.720 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.255 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:17.732 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.297 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:17.773 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.320 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:17.826 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:32:59.321 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:17.827 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.117 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:18.594 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.118 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:18.595 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.137 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:18.613 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.460 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:18.966 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.461 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:18.967 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.476 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:18.982 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.689 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:19.194 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.690 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:19.195 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:00.701 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:19.207 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.115 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:38.621 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.128 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:38.634 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.174 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:38.651 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.180 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:38.657 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.210 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:38.715 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.212 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:38.717 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.217 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:38.723 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.244 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:38.720 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:20.278 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:38.784 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:23.577 INFO:tasks.workunit.client.0.smithi064.stdout:adosAio.RoundTripAppendPP (3455 ms) 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2572 ms) 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (4177 ms) 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (4911 ms) 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: setting pool EIO 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: max_success 0, min_failed 1 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5112 ms) 2024-04-29T20:33:23.578 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4999 ms) 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (374805 ms total) 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3435 ms) 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10056 ms) 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17280 ms) 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3101 ms) 2024-04-29T20:33:23.579 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33872 ms total) 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3862 ms) 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3862 ms total) 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (16077 ms) 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6672 ms) 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7537 ms) 2024-04-29T20:33:23.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7103 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3531 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7058 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7458 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6996 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7639 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7346 ms) 2024-04-29T20:33:23.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-29T20:33:23.582 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7925 ms) 2024-04-29T20:33:23.582 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-29T20:33:23.582 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7158 ms) 2024-04-29T20:33:23.582 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-29T20:33:23.582 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (8396 ms) 2024-04-29T20:33:39.975 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:58.480 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:39.978 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:58.484 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.025 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:58.528 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.032 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:58.508 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.033 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:58.509 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.037 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:58.543 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.040 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:58.546 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.078 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:58.555 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:33:40.096 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:58.601 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.293 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:18.798 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.299 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:18.805 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.350 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:18.856 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.359 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:18.836 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.365 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:18.842 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.371 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:18.876 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.372 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:18.878 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.415 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:18.892 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:00.425 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:18.931 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2443 ms) 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7823 ms) 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7847 ms) 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7713 ms) 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6847 ms) 2024-04-29T20:34:10.580 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7180 ms) 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7146 ms) 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (147896 ms total) 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (560435 ms total) 2024-04-29T20:34:10.581 INFO:tasks.workunit.client.0.smithi064.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-29T20:34:20.508 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:39.013 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.511 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:39.016 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.554 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:39.059 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.580 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:39.086 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.581 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:39.058 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.582 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:39.087 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.583 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:39.060 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.627 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:39.132 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:20.633 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:39.110 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.375 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:58.880 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.376 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:58.882 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.394 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:58.899 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.426 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:58.903 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.427 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:58.904 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.430 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:58.936 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.434 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:58.940 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.444 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:58.921 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:34:40.453 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:58.959 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.543 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:19.048 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.616 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:19.122 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.649 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:19.155 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.678 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:19.154 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.682 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:19.188 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.682 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:19.159 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.687 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:19.193 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.722 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:19.227 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:00.723 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:19.199 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.774 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:26.280 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.781 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:26.287 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.793 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:26.269 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.795 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:26.271 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.798 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:26.304 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.808 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:26.314 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.820 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:26.326 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.878 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:26.355 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:07.914 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:26.420 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.558 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:34.063 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.561 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:34.067 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.580 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:34.085 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.608 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:34.085 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.609 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:34.086 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.627 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:34.104 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.636 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:34.142 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.637 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:34.143 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:15.656 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:34.162 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.525 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:39.031 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.578 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:39.084 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.597 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:39.103 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.627 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:39.103 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.630 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:39.107 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.631 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:39.137 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.635 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:39.141 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.646 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:39.123 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:20.652 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:39.158 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.735 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:54.240 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.740 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:54.246 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.769 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:54.275 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.800 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:54.277 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.801 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:54.307 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.802 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:54.278 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.803 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:54.309 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.829 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:54.306 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:35.839 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:54.345 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.631 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:59.108 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.632 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:59.137 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.633 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:59.110 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.636 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:59.142 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.647 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:13:59.124 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.650 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:13:59.156 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.692 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:59.198 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.695 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:59.201 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:40.709 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:13:59.215 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.787 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:09.293 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.791 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:09.296 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.821 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:09.327 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.835 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:09.311 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.837 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:09.314 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.851 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:09.357 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.852 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:09.358 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.860 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:09.336 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:50.876 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:09.381 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:35:52.033 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-29T20:35:52.033 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-29T20:35:52.033 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-29T20:35:52.033 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: seed 38434 2024-04-29T20:35:52.033 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (29644 ms) 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (75675 ms) 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (889 ms) 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (106208 ms total) 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (64978 ms) 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-29T20:35:52.034 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (91511 ms) 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (58473 ms) 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: my_snaps [3] 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: my_snaps [4,3] 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: promoting some heads 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: waiting for scrubs... 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: done waiting 2024-04-29T20:35:52.035 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (161274 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11035 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (11954 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8880 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10140 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10863 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8573 ms) 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-29T20:35:52.036 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10639 ms) 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (39132 ms) 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10989 ms) 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (9694 ms) 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15643 ms) 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (11499 ms) 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-29T20:35:52.037 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11020 ms) 2024-04-29T20:36:00.649 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:19.155 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.704 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:19.210 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.730 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:19.236 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.756 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:19.233 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.758 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:19.235 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.761 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:19.267 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.763 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:19.269 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.784 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:19.261 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:00.788 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:19.293 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.673 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:24.179 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.721 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:24.197 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.722 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:24.198 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.726 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:24.231 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.738 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:24.214 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.742 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:24.248 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.775 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:24.280 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.778 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:24.284 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:05.793 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:24.298 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: ok, hit_set contains 282:602f83fe:::foo:head 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9712 ms) 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg_num = 32 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 0 ls 1714422969,0 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 1 ls 1714422969,0 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 2 ls 1714422969,0 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 3 ls 1714422969,0 2024-04-29T20:36:14.621 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 4 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 5 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 6 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 7 ls 1714422970,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 8 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 9 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 10 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 11 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 12 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 13 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 14 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 15 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 16 ls 1714422969,0 2024-04-29T20:36:14.622 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 17 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 18 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 19 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 20 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 21 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 22 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 23 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 24 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 25 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 26 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 27 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 28 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 29 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 30 ls 1714422969,0 2024-04-29T20:36:14.623 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg 31 ls 1714422969,0 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: pg_num = 32 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6cac518f:::0:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:02547ec2:::1:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f905c69b:::2:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:cfc208b3:::3:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d83876eb:::4:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b29083e3:::5:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c4fdafeb:::6:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:5c6b0b28:::7:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:bd63b0f1:::8:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e960b815:::9:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:52ea6a34:::10:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:89d3ae78:::11:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:de5d7c5f:::12:head 2024-04-29T20:36:14.624 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:566253c9:::13:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:62a1935d:::14:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:863748b0:::15:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3958e169:::16:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4d4dabf9:::17:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:8391935d:::18:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:28883081:::19:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:69259c59:::20:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4bdb80b7:::21:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a11c5d71:::22:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:271af37b:::23:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:95b121be:::24:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:58d1031b:::25:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:0a050783:::26:head 2024-04-29T20:36:14.625 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c709704c:::27:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:cbe56eaf:::28:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:86b4b162:::29:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:70d89383:::30:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:dd450c7c:::31:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6d5729b1:::32:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c388f3fb:::33:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:56cfea31:::34:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9dbc1bf7:::35:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:40b74ccd:::36:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4d5aaf42:::37:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:920f362c:::38:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6cc53222:::39:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9cad833f:::40:head 2024-04-29T20:36:14.626 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:1ea84d41:::41:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c4480ef6:::42:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a694361e:::43:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d1bd33e9:::44:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ddc2cd5d:::45:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:2b782207:::46:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7b187fca:::47:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:90ecdf6f:::48:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a5ed95fe:::49:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ea0eaa55:::50:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f33ef17b:::51:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a0d1b2f6:::52:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:60c5229e:::53:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:edcbc575:::54:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:102cf253:::55:head 2024-04-29T20:36:14.627 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:efb7fb0b:::56:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:50d0a326:::57:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d4dc5daf:::58:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3a130462:::59:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ec87ed71:::60:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d5bc9454:::61:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3ddfe313:::62:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7c2816b9:::63:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:47e00e4d:::64:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c6410c18:::65:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b48ed237:::66:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:cd63ad31:::67:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b179e92b:::68:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:0d9f741a:::69:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6d3352ae:::70:head 2024-04-29T20:36:14.628 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c6d5c19e:::71:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:bc4729c3:::72:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:77e930b9:::73:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:0abeecfd:::74:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b7c37e15:::75:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b6378398:::76:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:02bd68de:::77:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:cc795d2d:::78:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:630d4fea:::79:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e0d29ef5:::80:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:fd6f13d2:::81:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:606461d5:::82:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:eadbdc43:::83:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:8761d0bb:::84:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9ef0186f:::85:head 2024-04-29T20:36:14.629 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e0d41294:::86:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:961de695:::87:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:1423148f:::88:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:633a8fa2:::89:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a8653809:::90:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3dac8b33:::91:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:35aad435:::92:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f6dcc343:::93:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:dbbdad87:::94:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:1cb48ce0:::95:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:03cd461c:::96:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:17a4ea99:::97:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9993c9a7:::98:head 2024-04-29T20:36:14.630 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6394211c:::99:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:94c7ae57:::100:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6fdee5bb:::101:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9a477fd1:::102:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:eb850916:::103:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:affc56b9:::104:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b42dc814:::105:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f319f8f0:::106:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9a40b9de:::107:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:8b524f28:::108:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e3de589f:::109:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:90f90a5b:::110:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a7b4f1d7:::111:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:af51766e:::112:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b6f90bd1:::113:head 2024-04-29T20:36:14.631 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e0261208:::114:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c9569ef7:::115:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:61bebe50:::116:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:fe93412b:::117:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d3d38bee:::118:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3100ba0c:::119:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d0560ada:::120:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f0ea8b35:::121:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:766f231a:::122:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a07a2582:::123:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:bd7c6b3a:::124:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:fb2ddaff:::125:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4408e1fe:::126:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ee1df7a7:::127:head 2024-04-29T20:36:14.632 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c3002909:::128:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4f48ffa9:::129:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:edf38733:::130:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c08425c0:::131:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:5f902d98:::132:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:41ea2c93:::133:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:813cee13:::134:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:0131818d:::135:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:26ba5a85:::136:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:381b8a5a:::137:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:28797e47:::138:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:bfca7f22:::139:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:36807075:::140:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:80b03975:::141:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:5c15709b:::142:head 2024-04-29T20:36:14.633 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f39ea15e:::143:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ea992956:::144:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:48887b1c:::145:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:9f24a9dd:::146:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:987f100b:::147:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d2dd3581:::148:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7fed1808:::149:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c80b70e9:::150:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:85ed90f9:::151:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:36428b24:::152:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d044c34a:::153:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7c18bf58:::154:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d1c21232:::155:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a7a3c575:::156:head 2024-04-29T20:36:14.634 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:87da0633:::157:head 2024-04-29T20:36:14.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d5ac3822:::158:head 2024-04-29T20:36:14.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3f20522d:::159:head 2024-04-29T20:36:14.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6ca26563:::160:head 2024-04-29T20:36:14.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:532ce135:::161:head 2024-04-29T20:36:14.635 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:c78863e6:::162:head 2024-04-29T20:36:20.991 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:39.497 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:20.993 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:39.499 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.018 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:39.523 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.042 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:39.518 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.043 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:39.520 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.063 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:39.540 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.064 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:39.570 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.066 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:39.572 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:21.112 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:39.617 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.024 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:54.529 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.029 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:54.535 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.070 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:54.576 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.077 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:54.553 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.078 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:54.555 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.098 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:54.603 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.099 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:54.605 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.102 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:54.579 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:36.125 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:54.631 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:40.956 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:59.461 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:40.961 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:59.467 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:40.971 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:14:59.477 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:41.008 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:59.484 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:41.009 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:59.485 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:41.018 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:14:59.495 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:41.027 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:59.533 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:41.029 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:59.535 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:41.039 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:14:59.545 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.131 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:14.636 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.137 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:14.643 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.178 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:14.683 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.188 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:14.665 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.190 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:14.667 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.216 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:14.693 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.222 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:14.727 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.223 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:14.729 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:36:56.250 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:14.756 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.219 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:19.725 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.221 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:19.727 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.235 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:19.741 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.271 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:19.748 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.272 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:19.778 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.272 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:19.749 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.273 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:19.779 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.286 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:19.763 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:01.288 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:19.794 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.100 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:39.605 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.101 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:39.607 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.127 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:39.633 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.150 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:39.626 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.151 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:39.628 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.158 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:39.663 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.159 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:39.665 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.176 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:39.652 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:21.191 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:39.697 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7462ddf6:::163:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:5d165639:::164:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f43765fc:::165:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b4c720e9:::166:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e694b040:::167:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:afa38db2:::168:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:77ba9f53:::169:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:87495034:::170:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7c96bf0e:::171:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:dbe346cc:::172:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:e943ec24:::173:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f97a9c0c:::174:head 2024-04-29T20:37:24.461 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:6f26e74d:::175:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4f95e106:::176:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:0e6f2f8f:::177:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:05db05f1:::178:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:38a78d66:::179:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:d095610b:::180:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:a1a9d709:::181:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:1e5d39db:::182:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:f7df4fb9:::183:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:03a7f161:::184:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ba70721e:::185:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:28e5662d:::186:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:973d52de:::187:head 2024-04-29T20:37:24.462 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:4303eb1c:::188:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:b990b48e:::189:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:29b8165b:::190:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:3547f197:::191:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:7e260936:::192:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:1abec7b1:::193:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:10fdda93:::194:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:15817eea:::195:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:770bab57:::196:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:ed9e13e7:::197:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:71471a8f:::198:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: checking for 283:10fb1d02:::199:head 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21728 ms) 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,0 2024-04-29T20:37:24.463 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: first is 1714422992 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,1714422997,1714422998,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,1714422997,1714422998,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,1714422997,1714422998,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,1714422997,1714422998,1714423000,1714423001,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,1714422997,1714422998,1714423000,1714423001,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422992,1714422994,1714422995,1714422997,1714422998,1714423000,1714423001,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714422995,1714422997,1714422998,1714423000,1714423001,1714423003,1714423004,0 2024-04-29T20:37:24.464 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: first now 1714422995, trimmed 2024-04-29T20:37:24.465 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23548 ms) 2024-04-29T20:37:24.465 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-29T20:37:24.465 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: foo0 2024-04-29T20:37:24.465 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-29T20:37:24.465 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (16692 ms) 2024-04-29T20:37:24.465 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-29T20:37:26.112 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:44.588 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.112 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:44.618 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.114 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:44.590 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.117 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:44.623 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.133 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:44.609 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.149 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:44.655 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.185 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:44.691 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.187 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:44.692 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:26.206 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:44.711 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.242 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:59.747 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.243 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:59.749 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.274 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:15:59.780 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.291 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:59.767 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.292 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:59.768 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.308 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:59.814 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.310 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:59.816 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.317 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:15:59.793 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:41.337 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:15:59.843 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.255 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:04.761 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.257 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:04.763 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.272 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:04.778 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.305 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:04.782 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.313 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:04.818 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.313 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:04.819 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.316 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:04.793 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.325 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:04.830 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:46.326 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:04.803 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.380 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:14.885 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.382 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:14.888 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.386 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:14.892 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.393 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:14.898 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.414 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:14.920 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.424 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:14.930 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.448 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:14.924 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.449 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:14.926 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:37:56.473 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:14.949 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.617 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:25.123 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.687 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:25.192 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.729 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:25.234 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.752 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:25.228 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.756 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:25.232 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.756 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:25.262 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.762 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:25.268 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.781 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:25.258 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:06.795 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:25.300 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.659 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:35.165 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.668 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:35.174 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.706 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:35.212 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.722 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:35.198 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.724 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:35.201 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.748 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:35.254 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.750 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:35.227 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.754 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:35.260 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:16.795 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:35.301 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.570 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:45.076 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.574 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:45.079 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.613 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:45.118 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.621 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:45.097 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.622 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:45.099 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.638 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:45.143 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.643 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:45.149 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.644 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:45.121 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:26.679 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:45.185 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.687 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:50.192 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.691 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:50.197 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.706 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:16:50.212 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.742 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:50.248 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.745 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:50.222 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.746 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:50.251 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.747 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:50.224 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.758 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:16:50.264 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:31.759 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:16:50.235 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.799 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:05.304 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.802 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:05.307 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.820 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:05.325 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.851 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:05.327 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.852 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:05.329 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.859 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:05.365 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.860 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:05.366 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.868 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:05.344 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:46.886 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:05.392 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.671 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:10.176 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.674 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:10.180 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.690 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:10.196 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.721 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:10.197 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.725 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:10.201 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.727 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:10.233 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.729 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:10.235 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.737 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:10.213 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:38:51.745 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:10.250 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.900 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:25.405 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.902 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:25.407 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.925 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:25.431 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.950 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:25.426 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.951 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:25.428 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.954 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:25.460 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.956 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:25.461 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.972 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:25.449 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:06.979 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:25.484 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:21.947 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:40.453 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.022 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:40.527 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.059 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:40.565 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.089 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:40.594 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.093 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:40.570 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.097 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:40.574 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.102 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:40.607 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.132 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:40.608 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:22.141 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:40.647 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.855 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:45.332 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.858 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:45.334 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.863 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:45.369 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.868 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:45.374 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.877 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:45.354 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.894 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:45.400 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.946 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:45.452 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.953 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:45.459 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:26.995 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:45.500 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.865 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:55.371 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.869 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:55.375 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.884 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:17:55.390 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.917 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:55.394 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.918 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:55.395 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.926 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:55.432 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.928 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:55.434 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.934 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:17:55.410 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:36.947 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:17:55.453 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.891 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:05.396 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.893 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:05.399 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.905 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:05.411 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.944 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:05.420 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.945 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:05.451 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.945 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:05.422 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.946 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:05.452 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.957 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:05.434 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:46.959 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:05.465 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.471 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:14.976 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.474 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:14.980 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.498 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:15.004 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.526 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:15.002 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.528 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:15.005 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.550 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:15.027 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.558 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:15.064 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.560 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:15.066 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:56.596 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:15.101 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.369 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:15.846 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.370 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:15.847 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.379 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:15.856 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.434 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:15.940 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.435 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:15.941 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.452 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:15.958 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.456 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:15.962 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.457 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:15.963 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:39:57.467 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:15.973 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.173 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:25.679 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.177 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:25.683 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.199 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:25.705 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.255 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:25.731 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.257 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:25.763 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.257 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:25.734 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.265 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:25.770 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.276 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:25.753 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:07.297 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:25.802 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:16.995 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:35.500 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.051 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:35.557 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.085 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:35.591 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.111 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:35.588 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.113 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:35.589 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.116 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:35.622 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.117 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:35.623 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.134 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:35.611 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:17.161 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:35.666 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.215 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:45.721 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.220 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:45.725 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.241 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:45.746 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.286 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:45.792 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.287 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:45.764 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.289 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:45.766 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.295 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:45.801 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.309 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:45.786 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:27.315 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:45.821 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.095 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:50.601 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.097 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:50.603 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.109 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:18:50.614 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.147 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:50.623 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.148 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:50.624 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.158 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:18:50.635 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.174 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:50.680 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.175 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:50.681 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:32.186 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:18:50.692 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:43.834 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:40:43.826+0000 26a93640 -1 snap_mapper.add_oid found existing snaps mapped on 301:6af3e344:test-rados-api-smithi064-38434-80::foo:2, removing 2024-04-29T20:40:43.839 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:40:43.824+0000 25290640 -1 snap_mapper.add_oid found existing snaps mapped on 301:6af3e344:test-rados-api-smithi064-38434-80::foo:2, removing 2024-04-29T20:40:47.311 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:05.817 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.328 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:05.833 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.330 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:05.807 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.342 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:05.848 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.347 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:05.852 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.414 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:05.890 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.467 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:05.973 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.498 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:06.003 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:47.519 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:05.996 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.164 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:10.670 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.167 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:10.672 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.182 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:10.687 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.215 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:10.692 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.216 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:10.693 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.232 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:10.709 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.239 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:10.744 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.244 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:10.749 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:40:52.260 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:10.766 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.244 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:25.749 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.247 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:25.753 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.281 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:25.787 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.295 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:25.772 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.300 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:25.776 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.302 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:25.807 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.303 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:25.809 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.328 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:25.805 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:07.336 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:25.842 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.074 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:26.551 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.075 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:26.552 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.084 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:26.561 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.135 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:26.641 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.136 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:26.642 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.141 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:26.647 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.142 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:26.648 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.146 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:26.652 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:08.152 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:26.658 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:20.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20748 ms) 2024-04-29T20:41:20.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-29T20:41:20.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25773 ms) 2024-04-29T20:41:20.008 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4434 ms) 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5689 ms) 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5323 ms) 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4692 ms) 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5612 ms) 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-29T20:41:20.009 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40279 ms) 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17941 ms) 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5565 ms) 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4624 ms) 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-29T20:41:20.010 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27322 ms) 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15124 ms) 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6245 ms) 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5518 ms) 2024-04-29T20:41:20.011 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: waiting for scrubs... 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: done waiting 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26256 ms) 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12362 ms) 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11486 ms) 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-29T20:41:20.012 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:41:20.013 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11302 ms) 2024-04-29T20:41:27.312 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:45.818 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.315 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:45.821 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.352 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:45.858 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.365 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:45.842 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.367 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:45.844 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.380 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:45.885 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.384 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:45.889 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.401 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:45.878 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:27.419 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:45.925 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.354 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:46.831 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.355 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:46.832 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.364 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:19:46.841 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.473 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:46.979 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.475 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:46.981 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.489 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:19:46.994 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.591 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:47.097 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.593 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:47.099 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:28.604 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:19:47.110 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.449 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:00.955 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.451 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:00.957 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.478 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:00.983 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.503 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:00.979 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.503 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:01.009 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.506 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:01.012 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.507 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:00.984 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.537 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:01.012 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:42.543 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:01.049 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.634 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:06.139 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.661 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:06.166 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.698 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:06.175 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.700 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:06.177 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.719 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:06.196 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.722 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:06.228 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.727 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:06.233 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.749 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:06.254 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:41:47.752 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:06.258 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.586 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:21.092 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.645 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:21.151 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.665 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:21.170 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.690 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:21.167 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.693 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:21.170 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.700 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:21.206 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.702 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:21.208 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.712 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:21.189 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:02.722 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:21.228 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.729 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:26.235 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.731 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:26.237 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.742 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:26.248 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.778 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:26.255 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.780 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:26.256 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.786 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:26.292 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.789 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:26.295 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.790 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:26.267 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:07.801 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:26.307 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.825 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:31.331 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.828 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:31.335 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.855 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:31.361 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.890 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:31.367 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.892 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:31.370 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.919 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:31.395 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.927 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:31.433 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.931 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:31.437 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:12.970 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:31.476 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:17.947 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:36.453 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:17.954 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:36.460 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:17.967 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:36.473 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:17.998 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:36.475 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:18.001 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:36.478 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:18.003 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:36.509 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:18.005 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:36.512 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:18.014 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:36.491 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:18.019 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:36.525 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:26.177 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:42:26.169+0000 26a93640 -1 snap_mapper.add_oid found existing snaps mapped on 103:67c0eb7a:test-rados-api-smithi064-38434-97::foo:21, removing 2024-04-29T20:42:26.188 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:42:26.151+0000 25290640 -1 snap_mapper.add_oid found existing snaps mapped on 103:67c0eb7a:test-rados-api-smithi064-38434-97::foo:21, removing 2024-04-29T20:42:27.030 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:45.536 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.035 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:45.541 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.061 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:45.567 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.085 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:45.562 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.102 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:45.607 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.102 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:45.608 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.105 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:45.582 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.120 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:45.626 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.120 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:45.597 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.776 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:46.282 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.781 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:46.287 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.782 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:46.259 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.786 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:46.292 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.787 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:46.293 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.789 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:46.266 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.841 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:46.317 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.847 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:46.353 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:27.849 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:46.356 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.864 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:56.370 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.869 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:56.375 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.905 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:20:56.412 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.923 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:56.400 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.924 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:56.402 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.936 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:56.442 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.941 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:56.447 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.948 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:20:56.425 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:37.964 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:20:56.470 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.904 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:06.410 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.906 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:06.412 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.944 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:06.450 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.954 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:06.431 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.955 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:06.432 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.974 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:06.480 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.976 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:06.482 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:47.979 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:06.456 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:48.001 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:06.507 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.921 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:11.427 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.923 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:11.429 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.938 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:11.444 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.972 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:11.449 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.973 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:11.451 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.978 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:11.484 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.980 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:11.487 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.987 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:11.465 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:42:52.996 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:11.502 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:07.977 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:26.483 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:07.979 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:26.485 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.011 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:26.517 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.034 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:26.511 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.035 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:26.513 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.038 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:26.545 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.042 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:26.548 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.064 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:26.541 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.094 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:26.600 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.736 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:27.213 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.737 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:27.214 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.746 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:27.223 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.921 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:27.427 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.922 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:27.428 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.960 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:27.466 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.998 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:27.504 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:08.999 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:27.506 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:09.016 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:27.522 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.019 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:41.525 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.021 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:41.527 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.053 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:41.558 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.072 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:41.549 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.073 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:41.580 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.075 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:41.582 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.091 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:41.569 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.113 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:41.619 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:23.120 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:41.598 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.300 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:46.806 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.308 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:46.814 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.335 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:46.812 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.338 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:46.815 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.342 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:46.849 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.346 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:46.853 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.357 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:46.863 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.377 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:46.854 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:28.422 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:46.928 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.070 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:56.576 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.073 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:56.580 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.114 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:21:56.621 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.119 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:56.596 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.124 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:56.601 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.137 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:56.643 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.141 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:56.647 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.149 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:21:56.626 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:38.173 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:21:56.679 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.125 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:06.631 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.127 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:06.633 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.164 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:06.670 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.173 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:06.651 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.175 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:06.652 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.198 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:06.704 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.198 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:06.675 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.204 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:06.711 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:48.229 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:06.735 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.411 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:11.918 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.413 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:11.920 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.429 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:11.936 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.460 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:11.937 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.461 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:11.939 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.469 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:11.976 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.471 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:11.977 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.477 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:11.954 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:43:53.487 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:11.993 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.277 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:26.783 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.281 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:26.788 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.310 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:26.816 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.323 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:26.800 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.324 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:26.831 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.328 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:26.805 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.330 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:26.836 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.356 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:26.833 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:08.361 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:26.868 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.058 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:27.535 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.059 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:27.536 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.068 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:27.546 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.169 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:27.676 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.170 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:27.677 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.173 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:27.680 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.175 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:27.681 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.181 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:27.687 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:09.185 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:27.691 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.323 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:41.829 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.328 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:41.834 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.345 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:41.852 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.379 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:41.886 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.381 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:41.887 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.381 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:41.858 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.382 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:41.859 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.402 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:41.909 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:23.403 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:41.881 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.435 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:46.941 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.437 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:46.944 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.450 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:46.956 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.486 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:46.963 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.487 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:46.964 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.487 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:46.994 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.491 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:46.997 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.501 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:46.978 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:28.503 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:47.009 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.943 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:52.449 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.944 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:52.451 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.945 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:52.422 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.951 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:52.429 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.969 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:52.476 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.972 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:52.449 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:33.999 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:52.506 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:34.001 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:52.508 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:34.032 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:52.539 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.523 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:57.029 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.529 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:57.036 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.544 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:22:57.051 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.587 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:57.064 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.591 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:57.068 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.607 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:57.113 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.611 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:22:57.088 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.613 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:57.120 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:38.630 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:22:57.136 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.557 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:07.063 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.561 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:07.068 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.586 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:07.093 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.609 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:07.086 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.611 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:07.089 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.615 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:07.122 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.618 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:07.125 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.637 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:07.114 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:48.651 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:07.157 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.612 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:12.118 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.680 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:12.187 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.701 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:12.208 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.736 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:12.213 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.741 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:12.218 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.756 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:12.263 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.757 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:12.234 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.763 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:12.270 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:44:53.790 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:12.296 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.573 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:27.079 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.576 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:27.083 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.610 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:27.116 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.629 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:27.106 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.631 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:27.109 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.635 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:27.141 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.639 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:27.145 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.662 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:27.140 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:08.673 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:27.180 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.386 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:27.863 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.387 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:27.865 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.395 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:27.873 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.438 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:27.944 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.438 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:27.945 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.453 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:27.960 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.457 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:27.964 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.458 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:27.965 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:09.467 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:27.974 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-04-29T20:45:23.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:45:23.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11824 ms) 2024-04-29T20:45:23.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-29T20:45:23.179 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7218 ms) 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6740 ms) 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25977 ms) 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (16302 ms) 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12597 ms) 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-29T20:45:23.180 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 00000012 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13675 ms) 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: require_osd_release = squid 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7849 ms) 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (976455 ms total) 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1392 ms) 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11069 ms) 2024-04-29T20:45:23.181 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (20592 ms) 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (33071 ms total) 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8863 ms) 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8647 ms) 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: waiting for scrub... 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: done waiting 2024-04-29T20:45:23.182 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27754 ms) 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10327 ms) 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9872 ms) 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9330 ms) 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11846 ms) 2024-04-29T20:45:23.183 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-29T20:45:23.834 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:42.341 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.840 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:42.347 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.891 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:42.398 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.910 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:42.388 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.912 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:42.419 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.913 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:42.391 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.916 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:42.423 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.944 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:42.422 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:23.958 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:42.465 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.757 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:47.264 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.765 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:47.272 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.782 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:47.289 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.806 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:47.284 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.810 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:47.286 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.823 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:47.330 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.823 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:47.301 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.824 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:47.331 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:28.846 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:47.353 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.761 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:57.267 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.767 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:57.274 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.771 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:57.278 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.773 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:57.250 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.775 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:57.253 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.790 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:57.297 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.855 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:23:57.333 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.873 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:23:57.380 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:38.882 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:23:57.389 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.820 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:07.327 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.911 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:07.418 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.946 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:07.453 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.964 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:07.442 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.968 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:07.446 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.982 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:07.489 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.989 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:07.495 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:48.992 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:07.470 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:49.024 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:07.531 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.732 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:12.239 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.736 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:12.243 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.751 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:12.258 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.783 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:12.261 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.785 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:12.263 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.788 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:12.295 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.790 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:12.297 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.798 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:12.276 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:45:53.805 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:12.312 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:08.939 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:27.445 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:08.941 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:27.449 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:08.982 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:27.490 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:08.997 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:27.475 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:09.002 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:27.480 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:09.008 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:27.515 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:09.015 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:27.522 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:09.040 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:27.518 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:09.054 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:27.561 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.906 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:42.413 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.915 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:42.423 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.956 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:42.463 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.967 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:42.445 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.969 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:42.476 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.973 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:42.481 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:23.974 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:42.452 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:24.004 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:42.482 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:24.027 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:42.534 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.896 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:47.403 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.898 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:47.405 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.915 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:47.422 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.951 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:47.429 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.951 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:47.458 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.954 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:47.432 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.955 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:47.462 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.970 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:47.477 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:28.972 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:47.450 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.932 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:57.439 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.933 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:57.441 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.956 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:57.463 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.983 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:57.461 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.984 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:57.491 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.984 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:57.463 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:38.986 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:57.493 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:39.009 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:57.487 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:39.014 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:57.522 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:39.896 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:58.374 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:39.897 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:58.375 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:39.912 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:24:58.390 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:40.025 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:58.532 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:40.026 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:58.533 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:40.041 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:24:58.548 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:40.054 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:58.561 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:40.055 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:58.562 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:40.070 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:24:58.577 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:48.967 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:07.474 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:48.972 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:07.479 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:48.986 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:07.494 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:49.019 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:07.527 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:49.021 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:07.499 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:49.024 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:07.502 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:49.024 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:07.531 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:49.037 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:07.515 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:49.038 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:07.545 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.012 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:17.519 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.013 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:17.520 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.033 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:17.541 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.062 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:17.540 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.063 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:17.541 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.080 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:17.588 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.082 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:17.560 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.082 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:17.589 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:46:59.101 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:17.608 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.182 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:27.689 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.237 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:27.744 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.261 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:27.768 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.288 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:27.795 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.302 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:27.781 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.304 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:27.782 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.322 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:27.829 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.328 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:27.806 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:09.357 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:27.864 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.221 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:32.698 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.222 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:32.700 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.226 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:32.733 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.234 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:32.741 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.241 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:32.719 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.270 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:32.777 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.305 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:32.812 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.311 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:32.818 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:14.335 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:32.842 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.153 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:47.660 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.155 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:47.663 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.175 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:47.682 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.209 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:47.687 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.210 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:47.688 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.213 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:47.720 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.214 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:47.721 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.228 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:47.707 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.237 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:47.744 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.924 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:48.402 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.925 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:48.403 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:29.934 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:25:48.413 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:30.022 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:48.529 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:30.023 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:48.530 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:30.037 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:25:48.544 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:30.041 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:48.549 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:30.042 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:48.550 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:30.051 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:25:48.559 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.219 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:02.726 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.226 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:02.733 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.256 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:02.763 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.270 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:02.748 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.275 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:02.782 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.275 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:02.753 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.278 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:02.785 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.305 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:02.784 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:44.315 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:02.823 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.348 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:07.855 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.352 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:07.859 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.364 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:07.871 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.400 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:07.878 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.401 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:07.879 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.406 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:07.913 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.407 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:07.915 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.413 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:07.892 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:49.419 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:07.927 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.465 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:17.972 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.474 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:17.981 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.504 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:18.011 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.510 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:18.017 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.535 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:18.013 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.538 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:18.016 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.558 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:18.036 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.567 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:18.075 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:47:59.594 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:18.101 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.404 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:22.882 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.405 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:22.884 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.409 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:22.917 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.411 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:22.918 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.436 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:22.944 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.440 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:22.918 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.472 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:22.980 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.474 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:22.981 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:04.495 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:23.003 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.434 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:27.941 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.437 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:27.945 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.454 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:27.961 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.484 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:27.962 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.485 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:27.963 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.500 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:27.978 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.502 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:28.009 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.505 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:28.013 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:09.521 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:28.028 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.594 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:38.101 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.598 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:38.105 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.624 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:38.132 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.664 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:38.143 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.666 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:38.144 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.668 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:38.176 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.678 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:38.185 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.690 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:38.168 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:19.709 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:38.217 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:20.555 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10193 ms) 2024-04-29T20:48:20.555 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14464 ms) 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9947 ms) 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14793 ms) 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8967 ms) 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9825 ms) 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: ok, hit_set contains 333:602f83fe:::foo:head 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10041 ms) 2024-04-29T20:48:20.556 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: first is 1714423600 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,1714423605,1714423606,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,1714423605,1714423606,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,1714423605,1714423606,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,1714423605,1714423606,1714423609,1714423610,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,1714423605,1714423606,1714423609,1714423610,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423600,1714423602,1714423603,1714423605,1714423606,1714423609,1714423610,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: got ls 1714423603,1714423605,1714423606,1714423609,1714423610,1714423612,1714423613,0 2024-04-29T20:48:20.557 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: first now 1714423603, trimmed 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (23860 ms) 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: foo0 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18064 ms) 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20604 ms) 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24507 ms) 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5971 ms) 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4440 ms) 2024-04-29T20:48:20.558 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4573 ms) 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4516 ms) 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (271404 ms total) 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1410278 ms total) 2024-04-29T20:48:20.559 INFO:tasks.workunit.client.0.smithi064.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-29T20:48:20.563 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.563 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38137 2024-04-29T20:48:20.563 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38137 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38354 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38354 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38853 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38853 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38549 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38549 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38475 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38475 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38216 2024-04-29T20:48:20.564 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38216 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38532 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38532 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39025 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39025 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39035 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39035 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38241 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38241 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38897 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38897 2024-04-29T20:48:20.565 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38743 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38743 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38113 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38113 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38175 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38175 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.566 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38755 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38755 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38104 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38104 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38124 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38124 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39173 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39173 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38366 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38366 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.567 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39218 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39218 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38488 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38488 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39005 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39005 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39284 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39284 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38439 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38439 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38770 2024-04-29T20:48:20.568 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38770 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39132 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39132 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38940 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38940 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39194 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39194 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38684 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38684 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.569 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38334 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38334 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39237 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39237 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38581 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38581 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38097 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38097 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38153 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38153 2024-04-29T20:48:20.570 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38542 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38542 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38197 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38197 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38612 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38612 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=38641 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 38641 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.571 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39060 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39060 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ for t in "${!pids[@]}" 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ pid=39310 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ wait 39310 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ exit 0 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ cleanup 2024-04-29T20:48:20.572 INFO:tasks.workunit.client.0.smithi064.stderr:+ pkill -P 38091 2024-04-29T20:48:20.575 INFO:tasks.workunit.client.0.smithi064.stderr:+ true 2024-04-29T20:48:20.576 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-29T20:48:20.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-29T20:48:20.605 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-04-29T20:48:20.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-29T20:48:20.932 DEBUG:teuthology.parallel:result is None 2024-04-29T20:48:20.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-29T20:48:20.956 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-29T20:48:20.956 DEBUG:teuthology.orchestra.run.smithi064:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-29T20:48:21.022 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-29T20:48:21.022 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-29T20:48:21.031 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-29T20:48:21.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.183+0000 7f88e7681640 10 monclient: get_monmap_and_config 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.183+0000 7f88e7681640 10 monclient: build_initial_monmap 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.183+0000 7f88e7681640 10 monclient: monmap: 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:21.184264+0000 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:21.184264+0000 2024-04-29T20:48:21.183 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:21.184 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 Processor -- start 2024-04-29T20:48:21.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 -- start start 2024-04-29T20:48:21.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient: init 2024-04-29T20:48:21.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:21.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-29T20:48:21.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f88e0075280 0x7f88e0140500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): picked mon.noname-a con 0x7f88e0075280 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88e0075660 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f88e00b1b80 0x7f88e0140a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): picked mon.noname-c con 0x7f88e00b1b80 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e0070660 con 0x7f88e0075660 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f88e0070940 con 0x7f88e0075280 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f88e00b1370 con 0x7f88e00b1b80 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.184+0000 7f88e7681640 10 monclient(hunting): authenticate will time out at 3050.344523s 2024-04-29T20:48:21.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.185+0000 7f88e5c1e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f88e00b1b80 0x7f88e0140a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f88e00b1b80 0x7f88e0140a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:49796/0 (socket says 172.21.15.64:49796) 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 1 -- 172.21.15.64:0/848725839 learned_addr learned my addr 172.21.15.64:0/848725839 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): get_auth_request con 0x7f88e00b1b80 auth_method 0 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.186+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_done global_id 22111 payload 274 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 1 -- 172.21.15.64:0/848725839 >> v2:172.21.15.64:3300/0 conn(0x7f88e0075280 msgr2=0x7f88e0140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 1 --2- 172.21.15.64:0/848725839 >> v2:172.21.15.64:3300/0 conn(0x7f88e0075280 0x7f88e0140500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 1 -- 172.21.15.64:0/848725839 >> v1:172.21.15.64:6789/0 conn(0x7f88e0075660 legacy=0x7f88e0146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 10 monclient: _finish_hunting 0 2024-04-29T20:48:21.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 1 monclient: found mon.noname-c 2024-04-29T20:48:21.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:21.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.188+0000 7f88e5c1e640 1 -- 172.21.15.64:0/848725839 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e006f780 con 0x7f88e00b1b80 2024-04-29T20:48:21.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.190+0000 7f88e5c1e640 1 --2- 172.21.15.64:0/848725839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0140a40 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 crypto rx=0x7f88e006fd70 tx=0x7f88d802bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3143b007e0e0ab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 1 -- 172.21.15.64:0/848725839 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d802c960 con 0x7f88e00b1b80 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: dump: 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.194 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: _finish_auth 0 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: _check_auth_tickets 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 1 -- 172.21.15.64:0/848725839 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d802cb00 con 0x7f88e00b1b80 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88e7681640 5 monclient: authenticate success, global_id 22111 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88e7681640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88e7681640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88e7681640 4 set_mon_vals no callback set 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 1 -- 172.21.15.64:0/848725839 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80343f0 con 0x7f88e00b1b80 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:21.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.194+0000 7f88d7fff640 10 monclient: dump: 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.196 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 10 monclient: shutdown 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 -- 172.21.15.64:0/848725839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 msgr2=0x7f88e0140a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 --2- 172.21.15.64:0/848725839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0140a40 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 crypto rx=0x7f88e006fd70 tx=0x7f88d802bdc0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 -- 172.21.15.64:0/848725839 shutdown_connections 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 --2- 172.21.15.64:0/848725839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0140a40 unknown :-1 s=CLOSED pgs=5612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 --2- 172.21.15.64:0/848725839 >> v2:172.21.15.64:3300/0 conn(0x7f88e0075280 0x7f88e0140500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 -- 172.21.15.64:0/848725839 >> 172.21.15.64:0/848725839 conn(0x7f88e00b58e0 msgr2=0x7f88e00b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 -- 172.21.15.64:0/848725839 shutdown_connections 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 1 -- 172.21.15.64:0/848725839 wait complete. 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 10 monclient: build_initial_monmap 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.195+0000 7f88e7681640 10 monclient: monmap: 2024-04-29T20:48:21.199 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 1 librados: starting msgr at 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 1 librados: starting objecter 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 1 Processor -- start 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 1 -- start start 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 1 librados: setting wanted keys 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 1 librados: calling monclient init 2024-04-29T20:48:21.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.196+0000 7f88e7681640 10 monclient: init 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88e0075280 0x7f88e0142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): picked mon.noname-a con 0x7f88e0075280 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): picked mon.noname-b con 0x7f88e00b1b80 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f88e0141480 0x7f88e013ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): picked mon.noname-c con 0x7f88e0141480 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f88e0147590 con 0x7f88e0075280 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f88e0147410 con 0x7f88e00b1b80 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f88e014de00 con 0x7f88e0141480 2024-04-29T20:48:21.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.197+0000 7f88e7681640 10 monclient(hunting): authenticate will time out at 3050.356986s 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e5c1e640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f88e0141480 0x7f88e013ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0142810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:49800/0 (socket says 172.21.15.64:49800) 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 1 -- 172.21.15.64:0/2598403054 learned_addr learned my addr 172.21.15.64:0/2598403054 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): get_auth_request con 0x7f88e00b1b80 auth_method 0 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.198+0000 7f88e4c1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.199+0000 7f88e5c1e640 10 monclient(hunting): get_auth_request con 0x7f88e0141480 auth_method 0 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.199+0000 7f88e5c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.199+0000 7f88e5c1e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.199+0000 7f88e5c1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e541d640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88e0075280 0x7f88e0142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e541d640 10 monclient(hunting): get_auth_request con 0x7f88e0075280 auth_method 0 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e541d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e541d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e541d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.200+0000 7f88e5c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 10 monclient(hunting): handle_auth_done global_id 22114 payload 824 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 1 -- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f88e0141480 msgr2=0x7f88e013ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f88e0141480 0x7f88e013ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 1 -- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88e0075280 msgr2=0x7f88e0142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88e0075280 0x7f88e0142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 10 monclient: _finish_hunting 0 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 1 monclient: found mon.noname-b 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.201+0000 7f88e4c1c640 1 -- 172.21.15.64:0/2598403054 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e0143490 con 0x7f88e00b1b80 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.203+0000 7f88e4c1c640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0142810 secure :-1 s=READY pgs=5613 cs=0 l=1 rev1=1 crypto rx=0x7f88d0009fd0 tx=0x7f88d000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.203+0000 7f88d5ffb640 10 client.?.objecter ms_handle_connect 0x7f88e00b1b80 2024-04-29T20:48:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.203+0000 7f88d5ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0015020 con 0x7f88e00b1b80 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: dump: 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.207 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: _finish_auth 0 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d0011830 con 0x7f88e00b1b80 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 5 monclient: authenticate success, global_id 22114 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 10 monclient: _renew_subs 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e013f150 con 0x7f88e00b1b80 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0011c00 con 0x7f88e00b1b80 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d5ffb640 10 monclient: dump: 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.208 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 10 client.22114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 10 monclient: _renew_subs 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88d77fe640 4 set_mon_vals no callback set 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e01dbe90 con 0x7f88e00b1b80 2024-04-29T20:48:21.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.207+0000 7f88e7681640 1 librados: init done 2024-04-29T20:48:21.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.208+0000 7f88e7681640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:21.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.208+0000 7f88e7681640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:21.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.208+0000 7f88e7681640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.208+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e004fce0 con 0x7f88e00b1b80 2024-04-29T20:48:21.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.463+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f88d0018020 con 0x7f88e00b1b80 2024-04-29T20:48:21.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.463+0000 7f88d5ffb640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f88c406e2d0 0x7f88c4070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.463+0000 7f88e541d640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f88c406e2d0 0x7f88c4070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.463+0000 7f88e541d640 10 monclient: get_auth_request con 0x7f88c406e2d0 auth_method 0 2024-04-29T20:48:21.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.464+0000 7f88e541d640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f88c406e2d0 0x7f88c4070790 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00b810 tx=0x7f88dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:21.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.464+0000 7f88d5ffb640 10 client.22114.objecter ms_handle_connect 0x7f88c406e2d0 2024-04-29T20:48:21.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.466+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(975..975 src has 638..975) ==== 6337+0+0 (secure 0 0 0) 0x7f88d0057dd0 con 0x7f88e00b1b80 2024-04-29T20:48:21.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.466+0000 7f88d5ffb640 10 client.22114.objecter ms_dispatch 0x7f88e0000b90 osd_map(975..975 src has 638..975) 2024-04-29T20:48:21.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.466+0000 7f88d5ffb640 3 client.22114.objecter handle_osd_map got epochs [975,975] > 0 2024-04-29T20:48:21.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.466+0000 7f88d5ffb640 3 client.22114.objecter handle_osd_map decoding full epoch 975 2024-04-29T20:48:21.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.466+0000 7f88d5ffb640 20 client.22114.objecter dump_active .. 0 homeless 2024-04-29T20:48:21.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.476+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88d001f020 con 0x7f88e00b1b80 2024-04-29T20:48:21.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.476+0000 7f88d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:21.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.476+0000 7f88d5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:21.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.671+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f88e0144cf0 con 0x7f88c406e2d0 2024-04-29T20:48:21.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.675+0000 7f88d5ffb640 1 -- 172.21.15.64:0/2598403054 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116282 (secure 0 0 0) 0x7f88e0144cf0 con 0x7f88c406e2d0 2024-04-29T20:48:21.679 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:48:21.680 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:48:21.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 10 librados: watch_flush enter 2024-04-29T20:48:21.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 10 librados: watch_flush exit 2024-04-29T20:48:21.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 20 client.22114.objecter shutdown clearing up homeless session... 2024-04-29T20:48:21.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 10 client.22114.objecter successfully canceled tick 2024-04-29T20:48:21.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f88c406e2d0 msgr2=0x7f88c4070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f88c406e2d0 0x7f88c4070790 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00b810 tx=0x7f88dc008040 comp rx=0 tx=0).stop 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 10 monclient: shutdown 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 msgr2=0x7f88e0142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0142810 secure :-1 s=READY pgs=5613 cs=0 l=1 rev1=1 crypto rx=0x7f88d0009fd0 tx=0x7f88d000b870 comp rx=0 tx=0).stop 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 shutdown_connections 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f88c406e2d0 0x7f88c4070790 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f88e0141480 0x7f88e013ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88e00b1b80 0x7f88e0142810 unknown :-1 s=CLOSED pgs=5613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 --2- 172.21.15.64:0/2598403054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88e0075280 0x7f88e0142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 >> 172.21.15.64:0/2598403054 conn(0x7f88e00b58e0 msgr2=0x7f88e00ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 shutdown_connections 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 -- 172.21.15.64:0/2598403054 wait complete. 2024-04-29T20:48:21.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.677+0000 7f88e7681640 1 librados: shutdown 2024-04-29T20:48:21.688 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2024-04-29T20:48:21.630542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":617804,"kb_used_data":44712,"kb_used_omap":12,"kb_used_meta":571379,"kb_avail":754356916,"statfs":{"total":773094113280,"available":772461481984,"internally_reserved":0,"allocated":45785088,"data_stored":23835494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":585092658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":565,"apply_latency_ms":565,"commit_latency_ns":565000000,"apply_latency_ns":565000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000717"},"pg_stats":[{"pgid":"11.1b","version":"0'0","reported_seq":1854,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.129142+0000","last_change":"2024-04-29T20:25:19.221101+0000","last_active":"2024-04-29T20:48:19.129142+0000","last_peered":"2024-04-29T20:48:19.129142+0000","last_clean":"2024-04-29T20:48:19.129142+0000","last_became_active":"2024-04-29T20:25:19.211321+0000","last_became_peered":"2024-04-29T20:25:19.211321+0000","last_unstale":"2024-04-29T20:48:19.129142+0000","last_undegraded":"2024-04-29T20:48:19.129142+0000","last_fullsized":"2024-04-29T20:48:19.129142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:42:34.869149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1846,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.552440+0000","last_change":"2024-04-29T20:25:24.951540+0000","last_active":"2024-04-29T20:48:19.552440+0000","last_peered":"2024-04-29T20:48:19.552440+0000","last_clean":"2024-04-29T20:48:19.552440+0000","last_became_active":"2024-04-29T20:25:24.924922+0000","last_became_peered":"2024-04-29T20:25:24.924922+0000","last_unstale":"2024-04-29T20:48:19.552440+0000","last_undegraded":"2024-04-29T20:48:19.552440+0000","last_fullsized":"2024-04-29T20:48:19.552440+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:01:14.186073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1847,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.548496+0000","last_change":"2024-04-29T20:25:25.197139+0000","last_active":"2024-04-29T20:48:19.548496+0000","last_peered":"2024-04-29T20:48:19.548496+0000","last_clean":"2024-04-29T20:48:19.548496+0000","last_became_active":"2024-04-29T20:25:25.191769+0000","last_became_peered":"2024-04-29T20:25:25.191769+0000","last_unstale":"2024-04-29T20:48:19.548496+0000","last_undegraded":"2024-04-29T20:48:19.548496+0000","last_fullsized":"2024-04-29T20:48:19.548496+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:57:41.743326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1853,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.515927+0000","last_change":"2024-04-29T20:25:26.047119+0000","last_active":"2024-04-29T20:48:18.515927+0000","last_peered":"2024-04-29T20:48:18.515927+0000","last_clean":"2024-04-29T20:48:18.515927+0000","last_became_active":"2024-04-29T20:25:26.041502+0000","last_became_peered":"2024-04-29T20:25:26.041502+0000","last_unstale":"2024-04-29T20:48:18.515927+0000","last_undegraded":"2024-04-29T20:48:18.515927+0000","last_fullsized":"2024-04-29T20:48:18.515927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:41:25.674969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1843,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-04-29T20:48:16.928208+0000","last_change":"2024-04-29T20:25:25.625395+0000","last_active":"2024-04-29T20:48:16.928208+0000","last_peered":"2024-04-29T20:48:16.928208+0000","last_clean":"2024-04-29T20:48:16.928208+0000","last_became_active":"2024-04-29T20:25:25.611019+0000","last_became_peered":"2024-04-29T20:25:25.611019+0000","last_unstale":"2024-04-29T20:48:16.928208+0000","last_undegraded":"2024-04-29T20:48:16.928208+0000","last_fullsized":"2024-04-29T20:48:16.928208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:57.199660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1864,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.925739+0000","last_change":"2024-04-29T20:25:22.691865+0000","last_active":"2024-04-29T20:48:17.925739+0000","last_peered":"2024-04-29T20:48:17.925739+0000","last_clean":"2024-04-29T20:48:17.925739+0000","last_became_active":"2024-04-29T20:25:22.685992+0000","last_became_peered":"2024-04-29T20:25:22.685992+0000","last_unstale":"2024-04-29T20:48:17.925739+0000","last_undegraded":"2024-04-29T20:48:17.925739+0000","last_fullsized":"2024-04-29T20:48:17.925739+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:23:46.806032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1859,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.196055+0000","last_change":"2024-04-29T20:25:26.419912+0000","last_active":"2024-04-29T20:48:19.196055+0000","last_peered":"2024-04-29T20:48:19.196055+0000","last_clean":"2024-04-29T20:48:19.196055+0000","last_became_active":"2024-04-29T20:25:26.414222+0000","last_became_peered":"2024-04-29T20:25:26.414222+0000","last_unstale":"2024-04-29T20:48:19.196055+0000","last_undegraded":"2024-04-29T20:48:19.196055+0000","last_fullsized":"2024-04-29T20:48:19.196055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:26:46.324672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1858,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.083584+0000","last_change":"2024-04-29T20:25:31.138205+0000","last_active":"2024-04-29T20:48:21.083584+0000","last_peered":"2024-04-29T20:48:21.083584+0000","last_clean":"2024-04-29T20:48:21.083584+0000","last_became_active":"2024-04-29T20:25:31.114951+0000","last_became_peered":"2024-04-29T20:25:31.114951+0000","last_unstale":"2024-04-29T20:48:21.083584+0000","last_undegraded":"2024-04-29T20:48:21.083584+0000","last_fullsized":"2024-04-29T20:48:21.083584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:57:40.889307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1859,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.557315+0000","last_change":"2024-04-29T20:25:27.559546+0000","last_active":"2024-04-29T20:48:19.557315+0000","last_peered":"2024-04-29T20:48:19.557315+0000","last_clean":"2024-04-29T20:48:19.557315+0000","last_became_active":"2024-04-29T20:25:27.553478+0000","last_became_peered":"2024-04-29T20:25:27.553478+0000","last_unstale":"2024-04-29T20:48:19.557315+0000","last_undegraded":"2024-04-29T20:48:19.557315+0000","last_fullsized":"2024-04-29T20:48:19.557315+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:36:09.125537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1866,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.794967+0000","last_change":"2024-04-29T20:25:06.859912+0000","last_active":"2024-04-29T20:48:17.794967+0000","last_peered":"2024-04-29T20:48:17.794967+0000","last_clean":"2024-04-29T20:48:17.794967+0000","last_became_active":"2024-04-29T20:25:06.850044+0000","last_became_peered":"2024-04-29T20:25:06.850044+0000","last_unstale":"2024-04-29T20:48:17.794967+0000","last_undegraded":"2024-04-29T20:48:17.794967+0000","last_fullsized":"2024-04-29T20:48:17.794967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:27:20.439378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"43'7","reported_seq":1883,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.073821+0000","last_change":"2024-04-29T20:25:09.449508+0000","last_active":"2024-04-29T20:48:21.073821+0000","last_peered":"2024-04-29T20:48:21.073821+0000","last_clean":"2024-04-29T20:48:21.073821+0000","last_became_active":"2024-04-29T20:25:09.413652+0000","last_became_peered":"2024-04-29T20:25:09.413652+0000","last_unstale":"2024-04-29T20:48:21.073821+0000","last_undegraded":"2024-04-29T20:48:21.073821+0000","last_fullsized":"2024-04-29T20:48:21.073821+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:51:59.474941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1858,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.519481+0000","last_change":"2024-04-29T20:25:21.119321+0000","last_active":"2024-04-29T20:48:18.519481+0000","last_peered":"2024-04-29T20:48:18.519481+0000","last_clean":"2024-04-29T20:48:18.519481+0000","last_became_active":"2024-04-29T20:25:21.109982+0000","last_became_peered":"2024-04-29T20:25:21.109982+0000","last_unstale":"2024-04-29T20:48:18.519481+0000","last_undegraded":"2024-04-29T20:48:18.519481+0000","last_fullsized":"2024-04-29T20:48:18.519481+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:58:45.347986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.793026+0000","last_change":"2024-04-29T20:25:04.146851+0000","last_active":"2024-04-29T20:48:17.793026+0000","last_peered":"2024-04-29T20:48:17.793026+0000","last_clean":"2024-04-29T20:48:17.793026+0000","last_became_active":"2024-04-29T20:25:04.139931+0000","last_became_peered":"2024-04-29T20:25:04.139931+0000","last_unstale":"2024-04-29T20:48:17.793026+0000","last_undegraded":"2024-04-29T20:48:17.793026+0000","last_fullsized":"2024-04-29T20:48:17.793026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:07:25.721846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":1881,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.132629+0000","last_change":"2024-04-29T20:22:57.765613+0000","last_active":"2024-04-29T20:48:19.132629+0000","last_peered":"2024-04-29T20:48:19.132629+0000","last_clean":"2024-04-29T20:48:19.132629+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:48:19.132629+0000","last_undegraded":"2024-04-29T20:48:19.132629+0000","last_fullsized":"2024-04-29T20:48:19.132629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_clean_scrub_stamp":"2024-04-29T20:22:42.263517+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:06:48.686106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.560572+0000","last_change":"2024-04-29T20:22:58.874338+0000","last_active":"2024-04-29T20:48:19.560572+0000","last_peered":"2024-04-29T20:48:19.560572+0000","last_clean":"2024-04-29T20:48:19.560572+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:48:19.560572+0000","last_undegraded":"2024-04-29T20:48:19.560572+0000","last_fullsized":"2024-04-29T20:48:19.560572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:59:51.304097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1858,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.546550+0000","last_change":"2024-04-29T20:25:07.978420+0000","last_active":"2024-04-29T20:48:19.546550+0000","last_peered":"2024-04-29T20:48:19.546550+0000","last_clean":"2024-04-29T20:48:19.546550+0000","last_became_active":"2024-04-29T20:25:07.961102+0000","last_became_peered":"2024-04-29T20:25:07.961102+0000","last_unstale":"2024-04-29T20:48:19.546550+0000","last_undegraded":"2024-04-29T20:48:19.546550+0000","last_fullsized":"2024-04-29T20:48:19.546550+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T00:27:36.921570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.539733+0000","last_change":"2024-04-29T20:22:58.155879+0000","last_active":"2024-04-29T20:48:19.539733+0000","last_peered":"2024-04-29T20:48:19.539733+0000","last_clean":"2024-04-29T20:48:19.539733+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:48:19.539733+0000","last_undegraded":"2024-04-29T20:48:19.539733+0000","last_fullsized":"2024-04-29T20:48:19.539733+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:40:44.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1862,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.517746+0000","last_change":"2024-04-29T20:25:16.331754+0000","last_active":"2024-04-29T20:48:18.517746+0000","last_peered":"2024-04-29T20:48:18.517746+0000","last_clean":"2024-04-29T20:48:18.517746+0000","last_became_active":"2024-04-29T20:25:16.326218+0000","last_became_peered":"2024-04-29T20:25:16.326218+0000","last_unstale":"2024-04-29T20:48:18.517746+0000","last_undegraded":"2024-04-29T20:48:18.517746+0000","last_fullsized":"2024-04-29T20:48:18.517746+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:34:14.827307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.721476+0000","last_change":"2024-04-29T20:22:58.100523+0000","last_active":"2024-04-29T20:48:17.721476+0000","last_peered":"2024-04-29T20:48:17.721476+0000","last_clean":"2024-04-29T20:48:17.721476+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:48:17.721476+0000","last_undegraded":"2024-04-29T20:48:17.721476+0000","last_fullsized":"2024-04-29T20:48:17.721476+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:31:35.619213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1864,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.532898+0000","last_change":"2024-04-29T20:25:06.176161+0000","last_active":"2024-04-29T20:48:18.532898+0000","last_peered":"2024-04-29T20:48:18.532898+0000","last_clean":"2024-04-29T20:48:18.532898+0000","last_became_active":"2024-04-29T20:25:06.170624+0000","last_became_peered":"2024-04-29T20:25:06.170624+0000","last_unstale":"2024-04-29T20:48:18.532898+0000","last_undegraded":"2024-04-29T20:48:18.532898+0000","last_fullsized":"2024-04-29T20:48:18.532898+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:38:50.008456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.549301+0000","last_change":"2024-04-29T20:22:58.456344+0000","last_active":"2024-04-29T20:48:19.549301+0000","last_peered":"2024-04-29T20:48:19.549301+0000","last_clean":"2024-04-29T20:48:19.549301+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:48:19.549301+0000","last_undegraded":"2024-04-29T20:48:19.549301+0000","last_fullsized":"2024-04-29T20:48:19.549301+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:02:31.378764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1859,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.792356+0000","last_change":"2024-04-29T20:25:19.733222+0000","last_active":"2024-04-29T20:48:17.792356+0000","last_peered":"2024-04-29T20:48:17.792356+0000","last_clean":"2024-04-29T20:48:17.792356+0000","last_became_active":"2024-04-29T20:25:19.725027+0000","last_became_peered":"2024-04-29T20:25:19.725027+0000","last_unstale":"2024-04-29T20:48:17.792356+0000","last_undegraded":"2024-04-29T20:48:17.792356+0000","last_fullsized":"2024-04-29T20:48:17.792356+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:51:22.081409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.860416+0000","last_change":"2024-04-29T20:22:58.466585+0000","last_active":"2024-04-29T20:48:17.860416+0000","last_peered":"2024-04-29T20:48:17.860416+0000","last_clean":"2024-04-29T20:48:17.860416+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:48:17.860416+0000","last_undegraded":"2024-04-29T20:48:17.860416+0000","last_fullsized":"2024-04-29T20:48:17.860416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:05:28.648611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.764121+0000","last_change":"2024-04-29T20:25:04.688244+0000","last_active":"2024-04-29T20:48:17.764121+0000","last_peered":"2024-04-29T20:48:17.764121+0000","last_clean":"2024-04-29T20:48:17.764121+0000","last_became_active":"2024-04-29T20:25:04.678852+0000","last_became_peered":"2024-04-29T20:25:04.678852+0000","last_unstale":"2024-04-29T20:48:17.764121+0000","last_undegraded":"2024-04-29T20:48:17.764121+0000","last_fullsized":"2024-04-29T20:48:17.764121+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:05:13.179776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.570994+0000","last_change":"2024-04-29T20:22:58.225857+0000","last_active":"2024-04-29T20:48:19.570994+0000","last_peered":"2024-04-29T20:48:19.570994+0000","last_clean":"2024-04-29T20:48:19.570994+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:48:19.570994+0000","last_undegraded":"2024-04-29T20:48:19.570994+0000","last_fullsized":"2024-04-29T20:48:19.570994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:16:31.016297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1868,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.103740+0000","last_change":"2024-04-29T20:25:05.054910+0000","last_active":"2024-04-29T20:48:21.103740+0000","last_peered":"2024-04-29T20:48:21.103740+0000","last_clean":"2024-04-29T20:48:21.103740+0000","last_became_active":"2024-04-29T20:25:05.045385+0000","last_became_peered":"2024-04-29T20:25:05.045385+0000","last_unstale":"2024-04-29T20:48:21.103740+0000","last_undegraded":"2024-04-29T20:48:21.103740+0000","last_fullsized":"2024-04-29T20:48:21.103740+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:53:43.926594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.568074+0000","last_change":"2024-04-29T20:22:59.053068+0000","last_active":"2024-04-29T20:48:19.568074+0000","last_peered":"2024-04-29T20:48:19.568074+0000","last_clean":"2024-04-29T20:48:19.568074+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:48:19.568074+0000","last_undegraded":"2024-04-29T20:48:19.568074+0000","last_fullsized":"2024-04-29T20:48:19.568074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:40:41.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1868,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.086257+0000","last_change":"2024-04-29T20:25:19.272741+0000","last_active":"2024-04-29T20:48:21.086257+0000","last_peered":"2024-04-29T20:48:21.086257+0000","last_clean":"2024-04-29T20:48:21.086257+0000","last_became_active":"2024-04-29T20:25:19.206413+0000","last_became_peered":"2024-04-29T20:25:19.206413+0000","last_unstale":"2024-04-29T20:48:21.086257+0000","last_undegraded":"2024-04-29T20:48:21.086257+0000","last_fullsized":"2024-04-29T20:48:21.086257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:16:10.608426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1869,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.592275+0000","last_change":"2024-04-29T20:22:59.008821+0000","last_active":"2024-04-29T20:48:19.592275+0000","last_peered":"2024-04-29T20:48:19.592275+0000","last_clean":"2024-04-29T20:48:19.592275+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:48:19.592275+0000","last_undegraded":"2024-04-29T20:48:19.592275+0000","last_fullsized":"2024-04-29T20:48:19.592275+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:05:01.384817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1853,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.143095+0000","last_change":"2024-04-29T20:25:19.599585+0000","last_active":"2024-04-29T20:48:19.143095+0000","last_peered":"2024-04-29T20:48:19.143095+0000","last_clean":"2024-04-29T20:48:19.143095+0000","last_became_active":"2024-04-29T20:25:19.589991+0000","last_became_peered":"2024-04-29T20:25:19.589991+0000","last_unstale":"2024-04-29T20:48:19.143095+0000","last_undegraded":"2024-04-29T20:48:19.143095+0000","last_fullsized":"2024-04-29T20:48:19.143095+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:46:47.268065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1845,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.192864+0000","last_change":"2024-04-29T20:25:24.677178+0000","last_active":"2024-04-29T20:48:19.192864+0000","last_peered":"2024-04-29T20:48:19.192864+0000","last_clean":"2024-04-29T20:48:19.192864+0000","last_became_active":"2024-04-29T20:25:24.662027+0000","last_became_peered":"2024-04-29T20:25:24.662027+0000","last_unstale":"2024-04-29T20:48:19.192864+0000","last_undegraded":"2024-04-29T20:48:19.192864+0000","last_fullsized":"2024-04-29T20:48:19.192864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:37:00.015485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1858,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.583482+0000","last_change":"2024-04-29T20:25:26.584778+0000","last_active":"2024-04-29T20:48:19.583482+0000","last_peered":"2024-04-29T20:48:19.583482+0000","last_clean":"2024-04-29T20:48:19.583482+0000","last_became_active":"2024-04-29T20:25:26.576214+0000","last_became_peered":"2024-04-29T20:25:26.576214+0000","last_unstale":"2024-04-29T20:48:19.583482+0000","last_undegraded":"2024-04-29T20:48:19.583482+0000","last_fullsized":"2024-04-29T20:48:19.583482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:20:55.927114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.840203+0000","last_change":"2024-04-29T20:25:02.740652+0000","last_active":"2024-04-29T20:48:17.840203+0000","last_peered":"2024-04-29T20:48:17.840203+0000","last_clean":"2024-04-29T20:48:17.840203+0000","last_became_active":"2024-04-29T20:25:02.734276+0000","last_became_peered":"2024-04-29T20:25:02.734276+0000","last_unstale":"2024-04-29T20:48:17.840203+0000","last_undegraded":"2024-04-29T20:48:17.840203+0000","last_fullsized":"2024-04-29T20:48:17.840203+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:34.054845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.929618+0000","last_change":"2024-04-29T20:25:06.026305+0000","last_active":"2024-04-29T20:48:17.929618+0000","last_peered":"2024-04-29T20:48:17.929618+0000","last_clean":"2024-04-29T20:48:17.929618+0000","last_became_active":"2024-04-29T20:25:06.011110+0000","last_became_peered":"2024-04-29T20:25:06.011110+0000","last_unstale":"2024-04-29T20:48:17.929618+0000","last_undegraded":"2024-04-29T20:48:17.929618+0000","last_fullsized":"2024-04-29T20:48:17.929618+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:51:47.148992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1866,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.720492+0000","last_change":"2024-04-29T20:25:02.267425+0000","last_active":"2024-04-29T20:48:17.720492+0000","last_peered":"2024-04-29T20:48:17.720492+0000","last_clean":"2024-04-29T20:48:17.720492+0000","last_became_active":"2024-04-29T20:25:02.256407+0000","last_became_peered":"2024-04-29T20:25:02.256407+0000","last_unstale":"2024-04-29T20:48:17.720492+0000","last_undegraded":"2024-04-29T20:48:17.720492+0000","last_fullsized":"2024-04-29T20:48:17.720492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:42.168617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1857,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.134309+0000","last_change":"2024-04-29T20:25:05.528506+0000","last_active":"2024-04-29T20:48:19.134309+0000","last_peered":"2024-04-29T20:48:19.134309+0000","last_clean":"2024-04-29T20:48:19.134309+0000","last_became_active":"2024-04-29T20:25:05.517205+0000","last_became_peered":"2024-04-29T20:25:05.517205+0000","last_unstale":"2024-04-29T20:48:19.134309+0000","last_undegraded":"2024-04-29T20:48:19.134309+0000","last_fullsized":"2024-04-29T20:48:19.134309+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:03:11.364243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1858,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.182427+0000","last_change":"2024-04-29T20:25:11.189601+0000","last_active":"2024-04-29T20:48:19.182427+0000","last_peered":"2024-04-29T20:48:19.182427+0000","last_clean":"2024-04-29T20:48:19.182427+0000","last_became_active":"2024-04-29T20:25:11.174336+0000","last_became_peered":"2024-04-29T20:25:11.174336+0000","last_unstale":"2024-04-29T20:48:19.182427+0000","last_undegraded":"2024-04-29T20:48:19.182427+0000","last_fullsized":"2024-04-29T20:48:19.182427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:12:54.605306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1857,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.578483+0000","last_change":"2024-04-29T20:25:09.447149+0000","last_active":"2024-04-29T20:48:19.578483+0000","last_peered":"2024-04-29T20:48:19.578483+0000","last_clean":"2024-04-29T20:48:19.578483+0000","last_became_active":"2024-04-29T20:25:09.429696+0000","last_became_peered":"2024-04-29T20:25:09.429696+0000","last_unstale":"2024-04-29T20:48:19.578483+0000","last_undegraded":"2024-04-29T20:48:19.578483+0000","last_fullsized":"2024-04-29T20:48:19.578483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:40.988418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1863,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.889835+0000","last_change":"2024-04-29T20:25:22.557015+0000","last_active":"2024-04-29T20:48:17.889835+0000","last_peered":"2024-04-29T20:48:17.889835+0000","last_clean":"2024-04-29T20:48:17.889835+0000","last_became_active":"2024-04-29T20:25:22.551405+0000","last_became_peered":"2024-04-29T20:25:22.551405+0000","last_unstale":"2024-04-29T20:48:17.889835+0000","last_undegraded":"2024-04-29T20:48:17.889835+0000","last_fullsized":"2024-04-29T20:48:17.889835+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:48:10.829532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1856,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.188229+0000","last_change":"2024-04-29T20:25:12.278620+0000","last_active":"2024-04-29T20:48:19.188229+0000","last_peered":"2024-04-29T20:48:19.188229+0000","last_clean":"2024-04-29T20:48:19.188229+0000","last_became_active":"2024-04-29T20:25:12.263826+0000","last_became_peered":"2024-04-29T20:25:12.263826+0000","last_unstale":"2024-04-29T20:48:19.188229+0000","last_undegraded":"2024-04-29T20:48:19.188229+0000","last_fullsized":"2024-04-29T20:48:19.188229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:32:24.000890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1867,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.101672+0000","last_change":"2024-04-29T20:25:09.882543+0000","last_active":"2024-04-29T20:48:21.101672+0000","last_peered":"2024-04-29T20:48:21.101672+0000","last_clean":"2024-04-29T20:48:21.101672+0000","last_became_active":"2024-04-29T20:25:09.849524+0000","last_became_peered":"2024-04-29T20:25:09.849524+0000","last_unstale":"2024-04-29T20:48:21.101672+0000","last_undegraded":"2024-04-29T20:48:21.101672+0000","last_fullsized":"2024-04-29T20:48:21.101672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:32:59.196210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280821,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81080,"kb_used_data":5452,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290760,"statfs":{"total":96636764160,"available":96553738240,"internally_reserved":0,"allocated":5582848,"data_stored":2832718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":98,"apply_latency_ms":98,"commit_latency_ns":98000000,"apply_latency_ns":98000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280821,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81904,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289936,"statfs":{"total":96636764160,"available":96552894464,"internally_reserved":0,"allocated":6086656,"data_stored":3342363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280820,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68604,"kb_used_data":5456,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303236,"statfs":{"total":96636764160,"available":96566513664,"internally_reserved":0,"allocated":5586944,"data_stored":2832723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":166,"apply_latency_ms":166,"commit_latency_ns":166000000,"apply_latency_ns":166000000},"alerts":[]},{"osd":6,"up_from":20,"seq":85899346231,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81604,"kb_used_data":5460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290236,"statfs":{"total":96636764160,"available":96553201664,"internally_reserved":0,"allocated":5591040,"data_stored":2854406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509748,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81296,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290544,"statfs":{"total":96636764160,"available":96553517056,"internally_reserved":0,"allocated":5623808,"data_stored":2887973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509749,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69036,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302804,"statfs":{"total":96636764160,"available":96566071296,"internally_reserved":0,"allocated":6033408,"data_stored":3294005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509750,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73244,"kb_used_data":5576,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298596,"statfs":{"total":96636764160,"available":96561762304,"internally_reserved":0,"allocated":5709824,"data_stored":2965285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81036,"kb_used_data":5440,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290804,"statfs":{"total":96636764160,"available":96553783296,"internally_reserved":0,"allocated":5570560,"data_stored":2826021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:48:21.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.791+0000 7fec36e09640 10 monclient: get_monmap_and_config 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.791+0000 7fec36e09640 10 monclient: build_initial_monmap 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.792+0000 7fec36e09640 10 monclient: monmap: 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:21.792862+0000 2024-04-29T20:48:21.792 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:21.792862+0000 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:21.793 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.792+0000 7fec36e09640 1 Processor -- start 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.792+0000 7fec36e09640 1 -- start start 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.792+0000 7fec36e09640 10 monclient: init 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fec300b9570 0x7fec30140720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): picked mon.noname-c con 0x7fec300b9570 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec300b3240 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec30146a90 0x7fec30146f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): picked mon.noname-b con 0x7fec30146a90 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec300abd80 con 0x7fec300b3240 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fec300b9950 con 0x7fec300b9570 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fec300aac70 con 0x7fec30146a90 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.793+0000 7fec36e09640 10 monclient(hunting): authenticate will time out at 3050.953291s 2024-04-29T20:48:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec353a6640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec30146a90 0x7fec30146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fec300b9570 0x7fec30140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fec300b9570 0x7fec30140720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:49812/0 (socket says 172.21.15.64:49812) 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 1 -- 172.21.15.64:0/3276430821 learned_addr learned my addr 172.21.15.64:0/3276430821 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 10 monclient(hunting): get_auth_request con 0x7fec300b9570 auth_method 0 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec34ba5640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec353a6640 10 monclient(hunting): get_auth_request con 0x7fec30146a90 auth_method 0 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec353a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec353a6640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.794+0000 7fec353a6640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.795+0000 7fec34ba5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.795+0000 7fec34ba5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.795+0000 7fec34ba5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.795+0000 7fec353a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.795+0000 7fec353a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.795+0000 7fec353a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 10 monclient(hunting): handle_auth_done global_id 16334 payload 274 2024-04-29T20:48:21.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 1 -- 172.21.15.64:0/3276430821 >> v2:172.21.15.167:3300/0 conn(0x7fec300b9570 msgr2=0x7fec30140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:21.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 1 --2- 172.21.15.64:0/3276430821 >> v2:172.21.15.167:3300/0 conn(0x7fec300b9570 0x7fec30140720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 1 -- 172.21.15.64:0/3276430821 >> v1:172.21.15.167:6789/0 conn(0x7fec300b3240 legacy=0x7fec30140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 10 monclient: _finish_hunting 0 2024-04-29T20:48:21.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 1 monclient: found mon.noname-b 2024-04-29T20:48:21.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:21.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:21.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.797+0000 7fec353a6640 1 -- 172.21.15.64:0/3276430821 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec300abff0 con 0x7fec30146a90 2024-04-29T20:48:21.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.799+0000 7fec353a6640 1 --2- 172.21.15.64:0/3276430821 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec30146a90 0x7fec30146f20 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7fec180097a0 tx=0x7fec1802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d63c8618e627221 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:21.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 1 -- 172.21.15.64:0/3276430821 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18034070 con 0x7fec30146a90 2024-04-29T20:48:21.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: dump: 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.804 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: _finish_auth 0 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 1 -- 172.21.15.64:0/3276430821 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec18030610 con 0x7fec30146a90 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec36e09640 5 monclient: authenticate success, global_id 16334 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec36e09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec36e09640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 1 -- 172.21.15.64:0/3276430821 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180309e0 con 0x7fec30146a90 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec36e09640 4 set_mon_vals no callback set 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.803+0000 7fec2f7fe640 10 monclient: dump: 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.807 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 10 monclient: shutdown 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 -- 172.21.15.64:0/3276430821 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec30146a90 msgr2=0x7fec30146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 --2- 172.21.15.64:0/3276430821 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec30146a90 0x7fec30146f20 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7fec180097a0 tx=0x7fec1802b960 comp rx=0 tx=0).stop 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 -- 172.21.15.64:0/3276430821 shutdown_connections 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 --2- 172.21.15.64:0/3276430821 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec30146a90 0x7fec30146f20 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 --2- 172.21.15.64:0/3276430821 >> v2:172.21.15.167:3300/0 conn(0x7fec300b9570 0x7fec30140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 -- 172.21.15.64:0/3276430821 >> 172.21.15.64:0/3276430821 conn(0x7fec3006cba0 msgr2=0x7fec3006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 -- 172.21.15.64:0/3276430821 shutdown_connections 2024-04-29T20:48:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.804+0000 7fec36e09640 1 -- 172.21.15.64:0/3276430821 wait complete. 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 10 monclient: build_initial_monmap 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 10 monclient: monmap: 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 1 librados: starting msgr at 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 1 librados: starting objecter 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 1 Processor -- start 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 1 -- start start 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 1 librados: setting wanted keys 2024-04-29T20:48:21.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 1 librados: calling monclient init 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.805+0000 7fec36e09640 10 monclient: init 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 0x7fec3012e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): picked mon.noname-b con 0x7fec300b3240 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec300b9570 0x7fec3013e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): picked mon.noname-c con 0x7fec300b9570 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec30146a90 0x7fec30144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:21.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): picked mon.noname-a con 0x7fec30146a90 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fec301475e0 con 0x7fec30146a90 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fec30147460 con 0x7fec300b3240 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fec30147760 con 0x7fec300b9570 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:21.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.806+0000 7fec36e09640 10 monclient(hunting): authenticate will time out at 3050.966171s 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 0x7fec3012e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec2ffff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec300b9570 0x7fec3013e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec353a6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec30146a90 0x7fec30144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 0x7fec3012e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:49816/0 (socket says 172.21.15.64:49816) 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 1 -- 172.21.15.64:0/23023272 learned_addr learned my addr 172.21.15.64:0/23023272 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): get_auth_request con 0x7fec300b3240 auth_method 0 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec2ffff640 10 monclient(hunting): get_auth_request con 0x7fec300b9570 auth_method 0 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec353a6640 10 monclient(hunting): get_auth_request con 0x7fec30146a90 auth_method 0 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec353a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec353a6640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:21.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec353a6640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.807+0000 7fec34ba5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.808+0000 7fec2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.808+0000 7fec2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.808+0000 7fec2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.808+0000 7fec353a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.808+0000 7fec353a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.808+0000 7fec353a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:21.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 10 monclient(hunting): handle_auth_done global_id 22120 payload 824 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 1 -- 172.21.15.64:0/23023272 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec300b9570 msgr2=0x7fec3013e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec300b9570 0x7fec3013e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 1 -- 172.21.15.64:0/23023272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec30146a90 msgr2=0x7fec30144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec30146a90 0x7fec30144de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 10 monclient: _finish_hunting 0 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 1 monclient: found mon.noname-b 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.810+0000 7fec34ba5640 1 -- 172.21.15.64:0/23023272 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3013ff10 con 0x7fec300b3240 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.812+0000 7fec34ba5640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 0x7fec3012e1e0 secure :-1 s=READY pgs=5615 cs=0 l=1 rev1=1 crypto rx=0x7fec2800efc0 tx=0x7fec2800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:21.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.812+0000 7fec2d7fa640 10 client.?.objecter ms_handle_connect 0x7fec300b3240 2024-04-29T20:48:21.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.812+0000 7fec2d7fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:21.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2800cda0 con 0x7fec300b3240 2024-04-29T20:48:21.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: dump: 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.826 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: _finish_auth 0 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: _check_auth_tickets 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec28019070 con 0x7fec300b3240 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28014680 con 0x7fec300b3240 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2d7fa640 10 monclient: dump: 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:21.828 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.816+0000 7fec2effd640 4 set_mon_vals no callback set 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 5 monclient: authenticate success, global_id 22120 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 10 monclient: _renew_subs 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec301449c0 con 0x7fec300b3240 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 10 client.22120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 10 monclient: _renew_subs 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec30145320 con 0x7fec300b3240 2024-04-29T20:48:21.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.817+0000 7fec36e09640 1 librados: init done 2024-04-29T20:48:21.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.818+0000 7fec36e09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:21.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.818+0000 7fec36e09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:21.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.818+0000 7fec36e09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:21.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:21.818+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec30145580 con 0x7fec300b3240 2024-04-29T20:48:22.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.070+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec28005d10 con 0x7fec300b3240 2024-04-29T20:48:22.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.070+0000 7fec2d7fa640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fec1006e280 0x7fec10070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.071+0000 7fec2ffff640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fec1006e280 0x7fec10070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.071+0000 7fec2ffff640 10 monclient: get_auth_request con 0x7fec1006e280 auth_method 0 2024-04-29T20:48:22.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.071+0000 7fec2ffff640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fec1006e280 0x7fec10070740 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fec300b43a0 tx=0x7fec20009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:22.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.071+0000 7fec2d7fa640 10 client.22120.objecter ms_handle_connect 0x7fec1006e280 2024-04-29T20:48:22.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.074+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(975..975 src has 638..975) ==== 6337+0+0 (secure 0 0 0) 0x7fec28058920 con 0x7fec300b3240 2024-04-29T20:48:22.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.074+0000 7fec2d7fa640 10 client.22120.objecter ms_dispatch 0x7fec30000b90 osd_map(975..975 src has 638..975) 2024-04-29T20:48:22.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.074+0000 7fec2d7fa640 3 client.22120.objecter handle_osd_map got epochs [975,975] > 0 2024-04-29T20:48:22.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.074+0000 7fec2d7fa640 3 client.22120.objecter handle_osd_map decoding full epoch 975 2024-04-29T20:48:22.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.074+0000 7fec2d7fa640 20 client.22120.objecter dump_active .. 0 homeless 2024-04-29T20:48:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.084+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec28010040 con 0x7fec300b3240 2024-04-29T20:48:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.084+0000 7fec2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.084+0000 7fec2d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:22.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.270+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec30145cc0 con 0x7fec1006e280 2024-04-29T20:48:22.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.274+0000 7fec2d7fa640 1 -- 172.21.15.64:0/23023272 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116282 (secure 0 0 0) 0x7fec30145cc0 con 0x7fec1006e280 2024-04-29T20:48:22.275 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:48:22.276 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 10 librados: watch_flush enter 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 10 librados: watch_flush exit 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 20 client.22120.objecter shutdown clearing up homeless session... 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 10 client.22120.objecter successfully canceled tick 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fec1006e280 msgr2=0x7fec10070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fec1006e280 0x7fec10070740 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fec300b43a0 tx=0x7fec20009040 comp rx=0 tx=0).stop 2024-04-29T20:48:22.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 10 monclient: shutdown 2024-04-29T20:48:22.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 msgr2=0x7fec3012e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 0x7fec3012e1e0 secure :-1 s=READY pgs=5615 cs=0 l=1 rev1=1 crypto rx=0x7fec2800efc0 tx=0x7fec2800c490 comp rx=0 tx=0).stop 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 shutdown_connections 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fec1006e280 0x7fec10070740 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fec300b9570 0x7fec3013e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec30146a90 0x7fec30144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 --2- 172.21.15.64:0/23023272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec300b3240 0x7fec3012e1e0 unknown :-1 s=CLOSED pgs=5615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 >> 172.21.15.64:0/23023272 conn(0x7fec3006cba0 msgr2=0x7fec300b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:22.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 shutdown_connections 2024-04-29T20:48:22.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 -- 172.21.15.64:0/23023272 wait complete. 2024-04-29T20:48:22.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.276+0000 7fec36e09640 1 librados: shutdown 2024-04-29T20:48:22.287 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2024-04-29T20:48:21.630542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":617804,"kb_used_data":44712,"kb_used_omap":12,"kb_used_meta":571379,"kb_avail":754356916,"statfs":{"total":773094113280,"available":772461481984,"internally_reserved":0,"allocated":45785088,"data_stored":23835494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":585092658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":565,"apply_latency_ms":565,"commit_latency_ns":565000000,"apply_latency_ns":565000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000717"},"pg_stats":[{"pgid":"11.1b","version":"0'0","reported_seq":1854,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.129142+0000","last_change":"2024-04-29T20:25:19.221101+0000","last_active":"2024-04-29T20:48:19.129142+0000","last_peered":"2024-04-29T20:48:19.129142+0000","last_clean":"2024-04-29T20:48:19.129142+0000","last_became_active":"2024-04-29T20:25:19.211321+0000","last_became_peered":"2024-04-29T20:25:19.211321+0000","last_unstale":"2024-04-29T20:48:19.129142+0000","last_undegraded":"2024-04-29T20:48:19.129142+0000","last_fullsized":"2024-04-29T20:48:19.129142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:42:34.869149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1846,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.552440+0000","last_change":"2024-04-29T20:25:24.951540+0000","last_active":"2024-04-29T20:48:19.552440+0000","last_peered":"2024-04-29T20:48:19.552440+0000","last_clean":"2024-04-29T20:48:19.552440+0000","last_became_active":"2024-04-29T20:25:24.924922+0000","last_became_peered":"2024-04-29T20:25:24.924922+0000","last_unstale":"2024-04-29T20:48:19.552440+0000","last_undegraded":"2024-04-29T20:48:19.552440+0000","last_fullsized":"2024-04-29T20:48:19.552440+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:01:14.186073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1847,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.548496+0000","last_change":"2024-04-29T20:25:25.197139+0000","last_active":"2024-04-29T20:48:19.548496+0000","last_peered":"2024-04-29T20:48:19.548496+0000","last_clean":"2024-04-29T20:48:19.548496+0000","last_became_active":"2024-04-29T20:25:25.191769+0000","last_became_peered":"2024-04-29T20:25:25.191769+0000","last_unstale":"2024-04-29T20:48:19.548496+0000","last_undegraded":"2024-04-29T20:48:19.548496+0000","last_fullsized":"2024-04-29T20:48:19.548496+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:57:41.743326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1853,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.515927+0000","last_change":"2024-04-29T20:25:26.047119+0000","last_active":"2024-04-29T20:48:18.515927+0000","last_peered":"2024-04-29T20:48:18.515927+0000","last_clean":"2024-04-29T20:48:18.515927+0000","last_became_active":"2024-04-29T20:25:26.041502+0000","last_became_peered":"2024-04-29T20:25:26.041502+0000","last_unstale":"2024-04-29T20:48:18.515927+0000","last_undegraded":"2024-04-29T20:48:18.515927+0000","last_fullsized":"2024-04-29T20:48:18.515927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:41:25.674969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1843,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-04-29T20:48:16.928208+0000","last_change":"2024-04-29T20:25:25.625395+0000","last_active":"2024-04-29T20:48:16.928208+0000","last_peered":"2024-04-29T20:48:16.928208+0000","last_clean":"2024-04-29T20:48:16.928208+0000","last_became_active":"2024-04-29T20:25:25.611019+0000","last_became_peered":"2024-04-29T20:25:25.611019+0000","last_unstale":"2024-04-29T20:48:16.928208+0000","last_undegraded":"2024-04-29T20:48:16.928208+0000","last_fullsized":"2024-04-29T20:48:16.928208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:57.199660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1864,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.925739+0000","last_change":"2024-04-29T20:25:22.691865+0000","last_active":"2024-04-29T20:48:17.925739+0000","last_peered":"2024-04-29T20:48:17.925739+0000","last_clean":"2024-04-29T20:48:17.925739+0000","last_became_active":"2024-04-29T20:25:22.685992+0000","last_became_peered":"2024-04-29T20:25:22.685992+0000","last_unstale":"2024-04-29T20:48:17.925739+0000","last_undegraded":"2024-04-29T20:48:17.925739+0000","last_fullsized":"2024-04-29T20:48:17.925739+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:23:46.806032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1859,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.196055+0000","last_change":"2024-04-29T20:25:26.419912+0000","last_active":"2024-04-29T20:48:19.196055+0000","last_peered":"2024-04-29T20:48:19.196055+0000","last_clean":"2024-04-29T20:48:19.196055+0000","last_became_active":"2024-04-29T20:25:26.414222+0000","last_became_peered":"2024-04-29T20:25:26.414222+0000","last_unstale":"2024-04-29T20:48:19.196055+0000","last_undegraded":"2024-04-29T20:48:19.196055+0000","last_fullsized":"2024-04-29T20:48:19.196055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:26:46.324672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1858,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.083584+0000","last_change":"2024-04-29T20:25:31.138205+0000","last_active":"2024-04-29T20:48:21.083584+0000","last_peered":"2024-04-29T20:48:21.083584+0000","last_clean":"2024-04-29T20:48:21.083584+0000","last_became_active":"2024-04-29T20:25:31.114951+0000","last_became_peered":"2024-04-29T20:25:31.114951+0000","last_unstale":"2024-04-29T20:48:21.083584+0000","last_undegraded":"2024-04-29T20:48:21.083584+0000","last_fullsized":"2024-04-29T20:48:21.083584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:57:40.889307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1859,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.557315+0000","last_change":"2024-04-29T20:25:27.559546+0000","last_active":"2024-04-29T20:48:19.557315+0000","last_peered":"2024-04-29T20:48:19.557315+0000","last_clean":"2024-04-29T20:48:19.557315+0000","last_became_active":"2024-04-29T20:25:27.553478+0000","last_became_peered":"2024-04-29T20:25:27.553478+0000","last_unstale":"2024-04-29T20:48:19.557315+0000","last_undegraded":"2024-04-29T20:48:19.557315+0000","last_fullsized":"2024-04-29T20:48:19.557315+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:36:09.125537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1866,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.794967+0000","last_change":"2024-04-29T20:25:06.859912+0000","last_active":"2024-04-29T20:48:17.794967+0000","last_peered":"2024-04-29T20:48:17.794967+0000","last_clean":"2024-04-29T20:48:17.794967+0000","last_became_active":"2024-04-29T20:25:06.850044+0000","last_became_peered":"2024-04-29T20:25:06.850044+0000","last_unstale":"2024-04-29T20:48:17.794967+0000","last_undegraded":"2024-04-29T20:48:17.794967+0000","last_fullsized":"2024-04-29T20:48:17.794967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:27:20.439378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"43'7","reported_seq":1883,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.073821+0000","last_change":"2024-04-29T20:25:09.449508+0000","last_active":"2024-04-29T20:48:21.073821+0000","last_peered":"2024-04-29T20:48:21.073821+0000","last_clean":"2024-04-29T20:48:21.073821+0000","last_became_active":"2024-04-29T20:25:09.413652+0000","last_became_peered":"2024-04-29T20:25:09.413652+0000","last_unstale":"2024-04-29T20:48:21.073821+0000","last_undegraded":"2024-04-29T20:48:21.073821+0000","last_fullsized":"2024-04-29T20:48:21.073821+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:51:59.474941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1858,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.519481+0000","last_change":"2024-04-29T20:25:21.119321+0000","last_active":"2024-04-29T20:48:18.519481+0000","last_peered":"2024-04-29T20:48:18.519481+0000","last_clean":"2024-04-29T20:48:18.519481+0000","last_became_active":"2024-04-29T20:25:21.109982+0000","last_became_peered":"2024-04-29T20:25:21.109982+0000","last_unstale":"2024-04-29T20:48:18.519481+0000","last_undegraded":"2024-04-29T20:48:18.519481+0000","last_fullsized":"2024-04-29T20:48:18.519481+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:58:45.347986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.793026+0000","last_change":"2024-04-29T20:25:04.146851+0000","last_active":"2024-04-29T20:48:17.793026+0000","last_peered":"2024-04-29T20:48:17.793026+0000","last_clean":"2024-04-29T20:48:17.793026+0000","last_became_active":"2024-04-29T20:25:04.139931+0000","last_became_peered":"2024-04-29T20:25:04.139931+0000","last_unstale":"2024-04-29T20:48:17.793026+0000","last_undegraded":"2024-04-29T20:48:17.793026+0000","last_fullsized":"2024-04-29T20:48:17.793026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:07:25.721846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":1881,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.132629+0000","last_change":"2024-04-29T20:22:57.765613+0000","last_active":"2024-04-29T20:48:19.132629+0000","last_peered":"2024-04-29T20:48:19.132629+0000","last_clean":"2024-04-29T20:48:19.132629+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:48:19.132629+0000","last_undegraded":"2024-04-29T20:48:19.132629+0000","last_fullsized":"2024-04-29T20:48:19.132629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_clean_scrub_stamp":"2024-04-29T20:22:42.263517+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:06:48.686106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.560572+0000","last_change":"2024-04-29T20:22:58.874338+0000","last_active":"2024-04-29T20:48:19.560572+0000","last_peered":"2024-04-29T20:48:19.560572+0000","last_clean":"2024-04-29T20:48:19.560572+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:48:19.560572+0000","last_undegraded":"2024-04-29T20:48:19.560572+0000","last_fullsized":"2024-04-29T20:48:19.560572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:59:51.304097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1858,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.546550+0000","last_change":"2024-04-29T20:25:07.978420+0000","last_active":"2024-04-29T20:48:19.546550+0000","last_peered":"2024-04-29T20:48:19.546550+0000","last_clean":"2024-04-29T20:48:19.546550+0000","last_became_active":"2024-04-29T20:25:07.961102+0000","last_became_peered":"2024-04-29T20:25:07.961102+0000","last_unstale":"2024-04-29T20:48:19.546550+0000","last_undegraded":"2024-04-29T20:48:19.546550+0000","last_fullsized":"2024-04-29T20:48:19.546550+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T00:27:36.921570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.539733+0000","last_change":"2024-04-29T20:22:58.155879+0000","last_active":"2024-04-29T20:48:19.539733+0000","last_peered":"2024-04-29T20:48:19.539733+0000","last_clean":"2024-04-29T20:48:19.539733+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:48:19.539733+0000","last_undegraded":"2024-04-29T20:48:19.539733+0000","last_fullsized":"2024-04-29T20:48:19.539733+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:40:44.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1862,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.517746+0000","last_change":"2024-04-29T20:25:16.331754+0000","last_active":"2024-04-29T20:48:18.517746+0000","last_peered":"2024-04-29T20:48:18.517746+0000","last_clean":"2024-04-29T20:48:18.517746+0000","last_became_active":"2024-04-29T20:25:16.326218+0000","last_became_peered":"2024-04-29T20:25:16.326218+0000","last_unstale":"2024-04-29T20:48:18.517746+0000","last_undegraded":"2024-04-29T20:48:18.517746+0000","last_fullsized":"2024-04-29T20:48:18.517746+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:34:14.827307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.721476+0000","last_change":"2024-04-29T20:22:58.100523+0000","last_active":"2024-04-29T20:48:17.721476+0000","last_peered":"2024-04-29T20:48:17.721476+0000","last_clean":"2024-04-29T20:48:17.721476+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:48:17.721476+0000","last_undegraded":"2024-04-29T20:48:17.721476+0000","last_fullsized":"2024-04-29T20:48:17.721476+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:31:35.619213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1864,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.532898+0000","last_change":"2024-04-29T20:25:06.176161+0000","last_active":"2024-04-29T20:48:18.532898+0000","last_peered":"2024-04-29T20:48:18.532898+0000","last_clean":"2024-04-29T20:48:18.532898+0000","last_became_active":"2024-04-29T20:25:06.170624+0000","last_became_peered":"2024-04-29T20:25:06.170624+0000","last_unstale":"2024-04-29T20:48:18.532898+0000","last_undegraded":"2024-04-29T20:48:18.532898+0000","last_fullsized":"2024-04-29T20:48:18.532898+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:38:50.008456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.549301+0000","last_change":"2024-04-29T20:22:58.456344+0000","last_active":"2024-04-29T20:48:19.549301+0000","last_peered":"2024-04-29T20:48:19.549301+0000","last_clean":"2024-04-29T20:48:19.549301+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:48:19.549301+0000","last_undegraded":"2024-04-29T20:48:19.549301+0000","last_fullsized":"2024-04-29T20:48:19.549301+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:02:31.378764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1859,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.792356+0000","last_change":"2024-04-29T20:25:19.733222+0000","last_active":"2024-04-29T20:48:17.792356+0000","last_peered":"2024-04-29T20:48:17.792356+0000","last_clean":"2024-04-29T20:48:17.792356+0000","last_became_active":"2024-04-29T20:25:19.725027+0000","last_became_peered":"2024-04-29T20:25:19.725027+0000","last_unstale":"2024-04-29T20:48:17.792356+0000","last_undegraded":"2024-04-29T20:48:17.792356+0000","last_fullsized":"2024-04-29T20:48:17.792356+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:51:22.081409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.860416+0000","last_change":"2024-04-29T20:22:58.466585+0000","last_active":"2024-04-29T20:48:17.860416+0000","last_peered":"2024-04-29T20:48:17.860416+0000","last_clean":"2024-04-29T20:48:17.860416+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:48:17.860416+0000","last_undegraded":"2024-04-29T20:48:17.860416+0000","last_fullsized":"2024-04-29T20:48:17.860416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:05:28.648611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.764121+0000","last_change":"2024-04-29T20:25:04.688244+0000","last_active":"2024-04-29T20:48:17.764121+0000","last_peered":"2024-04-29T20:48:17.764121+0000","last_clean":"2024-04-29T20:48:17.764121+0000","last_became_active":"2024-04-29T20:25:04.678852+0000","last_became_peered":"2024-04-29T20:25:04.678852+0000","last_unstale":"2024-04-29T20:48:17.764121+0000","last_undegraded":"2024-04-29T20:48:17.764121+0000","last_fullsized":"2024-04-29T20:48:17.764121+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:05:13.179776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.570994+0000","last_change":"2024-04-29T20:22:58.225857+0000","last_active":"2024-04-29T20:48:19.570994+0000","last_peered":"2024-04-29T20:48:19.570994+0000","last_clean":"2024-04-29T20:48:19.570994+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:48:19.570994+0000","last_undegraded":"2024-04-29T20:48:19.570994+0000","last_fullsized":"2024-04-29T20:48:19.570994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:16:31.016297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1868,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.103740+0000","last_change":"2024-04-29T20:25:05.054910+0000","last_active":"2024-04-29T20:48:21.103740+0000","last_peered":"2024-04-29T20:48:21.103740+0000","last_clean":"2024-04-29T20:48:21.103740+0000","last_became_active":"2024-04-29T20:25:05.045385+0000","last_became_peered":"2024-04-29T20:25:05.045385+0000","last_unstale":"2024-04-29T20:48:21.103740+0000","last_undegraded":"2024-04-29T20:48:21.103740+0000","last_fullsized":"2024-04-29T20:48:21.103740+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:53:43.926594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.568074+0000","last_change":"2024-04-29T20:22:59.053068+0000","last_active":"2024-04-29T20:48:19.568074+0000","last_peered":"2024-04-29T20:48:19.568074+0000","last_clean":"2024-04-29T20:48:19.568074+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:48:19.568074+0000","last_undegraded":"2024-04-29T20:48:19.568074+0000","last_fullsized":"2024-04-29T20:48:19.568074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:40:41.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1868,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.086257+0000","last_change":"2024-04-29T20:25:19.272741+0000","last_active":"2024-04-29T20:48:21.086257+0000","last_peered":"2024-04-29T20:48:21.086257+0000","last_clean":"2024-04-29T20:48:21.086257+0000","last_became_active":"2024-04-29T20:25:19.206413+0000","last_became_peered":"2024-04-29T20:25:19.206413+0000","last_unstale":"2024-04-29T20:48:21.086257+0000","last_undegraded":"2024-04-29T20:48:21.086257+0000","last_fullsized":"2024-04-29T20:48:21.086257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:16:10.608426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1869,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.592275+0000","last_change":"2024-04-29T20:22:59.008821+0000","last_active":"2024-04-29T20:48:19.592275+0000","last_peered":"2024-04-29T20:48:19.592275+0000","last_clean":"2024-04-29T20:48:19.592275+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:48:19.592275+0000","last_undegraded":"2024-04-29T20:48:19.592275+0000","last_fullsized":"2024-04-29T20:48:19.592275+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:05:01.384817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1853,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.143095+0000","last_change":"2024-04-29T20:25:19.599585+0000","last_active":"2024-04-29T20:48:19.143095+0000","last_peered":"2024-04-29T20:48:19.143095+0000","last_clean":"2024-04-29T20:48:19.143095+0000","last_became_active":"2024-04-29T20:25:19.589991+0000","last_became_peered":"2024-04-29T20:25:19.589991+0000","last_unstale":"2024-04-29T20:48:19.143095+0000","last_undegraded":"2024-04-29T20:48:19.143095+0000","last_fullsized":"2024-04-29T20:48:19.143095+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:46:47.268065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1845,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.192864+0000","last_change":"2024-04-29T20:25:24.677178+0000","last_active":"2024-04-29T20:48:19.192864+0000","last_peered":"2024-04-29T20:48:19.192864+0000","last_clean":"2024-04-29T20:48:19.192864+0000","last_became_active":"2024-04-29T20:25:24.662027+0000","last_became_peered":"2024-04-29T20:25:24.662027+0000","last_unstale":"2024-04-29T20:48:19.192864+0000","last_undegraded":"2024-04-29T20:48:19.192864+0000","last_fullsized":"2024-04-29T20:48:19.192864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:37:00.015485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1858,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.583482+0000","last_change":"2024-04-29T20:25:26.584778+0000","last_active":"2024-04-29T20:48:19.583482+0000","last_peered":"2024-04-29T20:48:19.583482+0000","last_clean":"2024-04-29T20:48:19.583482+0000","last_became_active":"2024-04-29T20:25:26.576214+0000","last_became_peered":"2024-04-29T20:25:26.576214+0000","last_unstale":"2024-04-29T20:48:19.583482+0000","last_undegraded":"2024-04-29T20:48:19.583482+0000","last_fullsized":"2024-04-29T20:48:19.583482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:20:55.927114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.840203+0000","last_change":"2024-04-29T20:25:02.740652+0000","last_active":"2024-04-29T20:48:17.840203+0000","last_peered":"2024-04-29T20:48:17.840203+0000","last_clean":"2024-04-29T20:48:17.840203+0000","last_became_active":"2024-04-29T20:25:02.734276+0000","last_became_peered":"2024-04-29T20:25:02.734276+0000","last_unstale":"2024-04-29T20:48:17.840203+0000","last_undegraded":"2024-04-29T20:48:17.840203+0000","last_fullsized":"2024-04-29T20:48:17.840203+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:34.054845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.929618+0000","last_change":"2024-04-29T20:25:06.026305+0000","last_active":"2024-04-29T20:48:17.929618+0000","last_peered":"2024-04-29T20:48:17.929618+0000","last_clean":"2024-04-29T20:48:17.929618+0000","last_became_active":"2024-04-29T20:25:06.011110+0000","last_became_peered":"2024-04-29T20:25:06.011110+0000","last_unstale":"2024-04-29T20:48:17.929618+0000","last_undegraded":"2024-04-29T20:48:17.929618+0000","last_fullsized":"2024-04-29T20:48:17.929618+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:51:47.148992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1866,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.720492+0000","last_change":"2024-04-29T20:25:02.267425+0000","last_active":"2024-04-29T20:48:17.720492+0000","last_peered":"2024-04-29T20:48:17.720492+0000","last_clean":"2024-04-29T20:48:17.720492+0000","last_became_active":"2024-04-29T20:25:02.256407+0000","last_became_peered":"2024-04-29T20:25:02.256407+0000","last_unstale":"2024-04-29T20:48:17.720492+0000","last_undegraded":"2024-04-29T20:48:17.720492+0000","last_fullsized":"2024-04-29T20:48:17.720492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:42.168617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1857,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.134309+0000","last_change":"2024-04-29T20:25:05.528506+0000","last_active":"2024-04-29T20:48:19.134309+0000","last_peered":"2024-04-29T20:48:19.134309+0000","last_clean":"2024-04-29T20:48:19.134309+0000","last_became_active":"2024-04-29T20:25:05.517205+0000","last_became_peered":"2024-04-29T20:25:05.517205+0000","last_unstale":"2024-04-29T20:48:19.134309+0000","last_undegraded":"2024-04-29T20:48:19.134309+0000","last_fullsized":"2024-04-29T20:48:19.134309+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:03:11.364243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1858,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.182427+0000","last_change":"2024-04-29T20:25:11.189601+0000","last_active":"2024-04-29T20:48:19.182427+0000","last_peered":"2024-04-29T20:48:19.182427+0000","last_clean":"2024-04-29T20:48:19.182427+0000","last_became_active":"2024-04-29T20:25:11.174336+0000","last_became_peered":"2024-04-29T20:25:11.174336+0000","last_unstale":"2024-04-29T20:48:19.182427+0000","last_undegraded":"2024-04-29T20:48:19.182427+0000","last_fullsized":"2024-04-29T20:48:19.182427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:12:54.605306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1857,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.578483+0000","last_change":"2024-04-29T20:25:09.447149+0000","last_active":"2024-04-29T20:48:19.578483+0000","last_peered":"2024-04-29T20:48:19.578483+0000","last_clean":"2024-04-29T20:48:19.578483+0000","last_became_active":"2024-04-29T20:25:09.429696+0000","last_became_peered":"2024-04-29T20:25:09.429696+0000","last_unstale":"2024-04-29T20:48:19.578483+0000","last_undegraded":"2024-04-29T20:48:19.578483+0000","last_fullsized":"2024-04-29T20:48:19.578483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:40.988418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1863,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.889835+0000","last_change":"2024-04-29T20:25:22.557015+0000","last_active":"2024-04-29T20:48:17.889835+0000","last_peered":"2024-04-29T20:48:17.889835+0000","last_clean":"2024-04-29T20:48:17.889835+0000","last_became_active":"2024-04-29T20:25:22.551405+0000","last_became_peered":"2024-04-29T20:25:22.551405+0000","last_unstale":"2024-04-29T20:48:17.889835+0000","last_undegraded":"2024-04-29T20:48:17.889835+0000","last_fullsized":"2024-04-29T20:48:17.889835+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:48:10.829532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1856,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.188229+0000","last_change":"2024-04-29T20:25:12.278620+0000","last_active":"2024-04-29T20:48:19.188229+0000","last_peered":"2024-04-29T20:48:19.188229+0000","last_clean":"2024-04-29T20:48:19.188229+0000","last_became_active":"2024-04-29T20:25:12.263826+0000","last_became_peered":"2024-04-29T20:25:12.263826+0000","last_unstale":"2024-04-29T20:48:19.188229+0000","last_undegraded":"2024-04-29T20:48:19.188229+0000","last_fullsized":"2024-04-29T20:48:19.188229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:32:24.000890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1867,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.101672+0000","last_change":"2024-04-29T20:25:09.882543+0000","last_active":"2024-04-29T20:48:21.101672+0000","last_peered":"2024-04-29T20:48:21.101672+0000","last_clean":"2024-04-29T20:48:21.101672+0000","last_became_active":"2024-04-29T20:25:09.849524+0000","last_became_peered":"2024-04-29T20:25:09.849524+0000","last_unstale":"2024-04-29T20:48:21.101672+0000","last_undegraded":"2024-04-29T20:48:21.101672+0000","last_fullsized":"2024-04-29T20:48:21.101672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:32:59.196210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280821,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81080,"kb_used_data":5452,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290760,"statfs":{"total":96636764160,"available":96553738240,"internally_reserved":0,"allocated":5582848,"data_stored":2832718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":98,"apply_latency_ms":98,"commit_latency_ns":98000000,"apply_latency_ns":98000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280821,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81904,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289936,"statfs":{"total":96636764160,"available":96552894464,"internally_reserved":0,"allocated":6086656,"data_stored":3342363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280820,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68604,"kb_used_data":5456,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303236,"statfs":{"total":96636764160,"available":96566513664,"internally_reserved":0,"allocated":5586944,"data_stored":2832723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":166,"apply_latency_ms":166,"commit_latency_ns":166000000,"apply_latency_ns":166000000},"alerts":[]},{"osd":6,"up_from":20,"seq":85899346231,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81604,"kb_used_data":5460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290236,"statfs":{"total":96636764160,"available":96553201664,"internally_reserved":0,"allocated":5591040,"data_stored":2854406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509748,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81296,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290544,"statfs":{"total":96636764160,"available":96553517056,"internally_reserved":0,"allocated":5623808,"data_stored":2887973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509749,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69036,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302804,"statfs":{"total":96636764160,"available":96566071296,"internally_reserved":0,"allocated":6033408,"data_stored":3294005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509750,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73244,"kb_used_data":5576,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298596,"statfs":{"total":96636764160,"available":96561762304,"internally_reserved":0,"allocated":5709824,"data_stored":2965285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81036,"kb_used_data":5440,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290804,"statfs":{"total":96636764160,"available":96553783296,"internally_reserved":0,"allocated":5570560,"data_stored":2826021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:48:22.289 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-29T20:48:22.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:48:22.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.390+0000 7fb3a558f640 10 monclient: get_monmap_and_config 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.390+0000 7fb3a558f640 10 monclient: build_initial_monmap 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.390+0000 7fb3a558f640 10 monclient: monmap: 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:22.391621+0000 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:22.391621+0000 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:22.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 1 Processor -- start 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 1 -- start start 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 10 monclient: init 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fb3a00b5fc0 0x7fb3a00aae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a00b5fc0 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb3a00b2ca0 0x7fb3a0146a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.391+0000 7fb3a558f640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a00b2ca0 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3a0140810 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:22.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a00ac030 con 0x7fb3a0140810 2024-04-29T20:48:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fb3a00b63a0 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fb3a00ac310 con 0x7fb3a00b2ca0 2024-04-29T20:48:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:22.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb3a558f640 10 monclient(hunting): authenticate will time out at 3051.551595s 2024-04-29T20:48:22.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb39e7fc640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fb3a00b2ca0 0x7fb3a0146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.392+0000 7fb39effd640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fb3a00b5fc0 0x7fb3a00aae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fb3a00b5fc0 0x7fb3a00aae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50234/0 (socket says 172.21.15.64:50234) 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 1 -- 172.21.15.64:0/3804241987 learned_addr learned my addr 172.21.15.64:0/3804241987 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): get_auth_request con 0x7fb3a00b5fc0 auth_method 0 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): get_auth_request con 0x7fb3a00b2ca0 auth_method 0 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.393+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 10 monclient(hunting): handle_auth_done global_id 14613 payload 274 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 1 -- 172.21.15.64:0/3804241987 >> v2:172.21.15.167:3300/0 conn(0x7fb3a00b2ca0 msgr2=0x7fb3a0146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 1 --2- 172.21.15.64:0/3804241987 >> v2:172.21.15.167:3300/0 conn(0x7fb3a00b2ca0 0x7fb3a0146a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 1 -- 172.21.15.64:0/3804241987 >> v1:172.21.15.167:6789/0 conn(0x7fb3a0140810 legacy=0x7fb3a00b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 10 monclient: _finish_hunting 0 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 1 monclient: found mon.noname-a 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:48:22.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.395+0000 7fb39effd640 1 -- 172.21.15.64:0/3804241987 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a00aa930 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.397+0000 7fb39effd640 1 --2- 172.21.15.64:0/3804241987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a00aae50 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00b37d0 tx=0x7fb39002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb068f5f0b487c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:22.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 1 -- 172.21.15.64:0/3804241987 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39002c960 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:22.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: dump: 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: _finish_auth 0 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: _check_auth_tickets 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 1 -- 172.21.15.64:0/3804241987 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb39002cb00 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb3a558f640 5 monclient: authenticate success, global_id 14613 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb3a558f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb3a558f640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 1 -- 172.21.15.64:0/3804241987 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3900343f0 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb3a558f640 4 set_mon_vals no callback set 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:48:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.400+0000 7fb39dffb640 10 monclient: dump: 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:22.403 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 10 monclient: shutdown 2024-04-29T20:48:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 -- 172.21.15.64:0/3804241987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a00b5fc0 msgr2=0x7fb3a00aae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 --2- 172.21.15.64:0/3804241987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a00aae50 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00b37d0 tx=0x7fb39002bdc0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 -- 172.21.15.64:0/3804241987 shutdown_connections 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 --2- 172.21.15.64:0/3804241987 >> v2:172.21.15.167:3300/0 conn(0x7fb3a00b2ca0 0x7fb3a0146a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 --2- 172.21.15.64:0/3804241987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a00aae50 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 -- 172.21.15.64:0/3804241987 >> 172.21.15.64:0/3804241987 conn(0x7fb3a0067900 msgr2=0x7fb3a0067d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 -- 172.21.15.64:0/3804241987 shutdown_connections 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.401+0000 7fb3a558f640 1 -- 172.21.15.64:0/3804241987 wait complete. 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.402+0000 7fb3a558f640 10 monclient: build_initial_monmap 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.402+0000 7fb3a558f640 10 monclient: monmap: 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.402+0000 7fb3a558f640 1 librados: starting msgr at 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.402+0000 7fb3a558f640 1 librados: starting objecter 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 Processor -- start 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 -- start start 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 librados: setting wanted keys 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 librados: calling monclient init 2024-04-29T20:48:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient: init 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb3a00b2ca0 0x7fb3a0145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a00b2ca0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a01d8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a00b5fc0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a0140810 0x7fb3a01d8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a0140810 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a0147110 con 0x7fb3a0140810 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a0146f90 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a0140ca0 con 0x7fb3a00b2ca0 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.403+0000 7fb3a558f640 10 monclient(hunting): authenticate will time out at 3051.563294s 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a0140810 0x7fb3a01d8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39effd640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb3a00b2ca0 0x7fb3a0145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39e7fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a01d8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a0140810 0x7fb3a01d8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50246/0 (socket says 172.21.15.64:50246) 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 1 -- 172.21.15.64:0/4259211397 learned_addr learned my addr 172.21.15.64:0/4259211397 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 10 monclient(hunting): get_auth_request con 0x7fb3a0140810 auth_method 0 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39e7fc640 10 monclient(hunting): get_auth_request con 0x7fb3a00b5fc0 auth_method 0 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39effd640 10 monclient(hunting): get_auth_request con 0x7fb3a00b2ca0 auth_method 0 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39effd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.404+0000 7fb39effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.405+0000 7fb39effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 10 monclient(hunting): handle_auth_done global_id 22126 payload 824 2024-04-29T20:48:22.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 1 -- 172.21.15.64:0/4259211397 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb3a00b2ca0 msgr2=0x7fb3a0145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb3a00b2ca0 0x7fb3a0145a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 1 -- 172.21.15.64:0/4259211397 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a0140810 msgr2=0x7fb3a01d8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a0140810 0x7fb3a01d8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 10 monclient: _finish_hunting 0 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 1 monclient: found mon.noname-b 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.407+0000 7fb39e7fc640 1 -- 172.21.15.64:0/4259211397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a0142360 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.410+0000 7fb39e7fc640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a01d8980 secure :-1 s=READY pgs=5617 cs=0 l=1 rev1=1 crypto rx=0x7fb38800e880 tx=0x7fb38800ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.410+0000 7fb3977fe640 10 client.?.objecter ms_handle_connect 0x7fb3a00b5fc0 2024-04-29T20:48:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.410+0000 7fb3977fe640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388017070 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: dump: 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:22.415 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: _finish_auth 0 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: _check_auth_tickets 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38800c940 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3a558f640 5 monclient: authenticate success, global_id 22126 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3a558f640 10 monclient: _renew_subs 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3a558f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a0142640 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38800cd30 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.413+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:22.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3977fe640 10 monclient: dump: 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb39cff9640 4 set_mon_vals no callback set 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3a558f640 10 client.22126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3a558f640 10 monclient: _renew_subs 2024-04-29T20:48:22.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3a558f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:22.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a0146090 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.414+0000 7fb3a558f640 1 librados: init done 2024-04-29T20:48:22.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.415+0000 7fb3a558f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:22.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.415+0000 7fb3a558f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:22.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.415+0000 7fb3a558f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:22.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.415+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a004fce0 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.433+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb38801a070 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.434+0000 7fb3977fe640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb38006e280 0x7fb380070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.434+0000 7fb39effd640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb38006e280 0x7fb380070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.434+0000 7fb39effd640 10 monclient: get_auth_request con 0x7fb38006e280 auth_method 0 2024-04-29T20:48:22.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.434+0000 7fb39effd640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb38006e280 0x7fb380070740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb390035000 tx=0x7fb390030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:22.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.434+0000 7fb3977fe640 10 client.22126.objecter ms_handle_connect 0x7fb38006e280 2024-04-29T20:48:22.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.436+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(975..975 src has 638..975) ==== 6337+0+0 (secure 0 0 0) 0x7fb3880580b0 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.436+0000 7fb3977fe640 10 client.22126.objecter ms_dispatch 0x7fb3a0000b90 osd_map(975..975 src has 638..975) 2024-04-29T20:48:22.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.436+0000 7fb3977fe640 3 client.22126.objecter handle_osd_map got epochs [975,975] > 0 2024-04-29T20:48:22.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.436+0000 7fb3977fe640 3 client.22126.objecter handle_osd_map decoding full epoch 975 2024-04-29T20:48:22.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.436+0000 7fb3977fe640 20 client.22126.objecter dump_active .. 0 homeless 2024-04-29T20:48:22.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.676+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb38805a010 con 0x7fb3a00b5fc0 2024-04-29T20:48:22.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.676+0000 7fb3977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:22.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.676+0000 7fb3977fe640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.865+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb3a01463b0 con 0x7fb38006e280 2024-04-29T20:48:22.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.868+0000 7fb3977fe640 1 -- 172.21.15.64:0/4259211397 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116282 (secure 0 0 0) 0x7fb3a01463b0 con 0x7fb38006e280 2024-04-29T20:48:22.869 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:48:22.870 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 10 librados: watch_flush enter 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 10 librados: watch_flush exit 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 20 client.22126.objecter shutdown clearing up homeless session... 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 10 client.22126.objecter successfully canceled tick 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb38006e280 msgr2=0x7fb380070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb38006e280 0x7fb380070740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb390035000 tx=0x7fb390030000 comp rx=0 tx=0).stop 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 10 monclient: shutdown 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3a00b5fc0 msgr2=0x7fb3a01d8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a01d8980 secure :-1 s=READY pgs=5617 cs=0 l=1 rev1=1 crypto rx=0x7fb38800e880 tx=0x7fb38800ed40 comp rx=0 tx=0).stop 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 shutdown_connections 2024-04-29T20:48:22.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fb38006e280 0x7fb380070740 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3a0140810 0x7fb3a01d8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3a00b5fc0 0x7fb3a01d8980 unknown :-1 s=CLOSED pgs=5617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 --2- 172.21.15.64:0/4259211397 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fb3a00b2ca0 0x7fb3a0145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 >> 172.21.15.64:0/4259211397 conn(0x7fb3a0067900 msgr2=0x7fb3a00b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 shutdown_connections 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 -- 172.21.15.64:0/4259211397 wait complete. 2024-04-29T20:48:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.871+0000 7fb3a558f640 1 librados: shutdown 2024-04-29T20:48:22.882 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2024-04-29T20:48:21.630542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":617804,"kb_used_data":44712,"kb_used_omap":12,"kb_used_meta":571379,"kb_avail":754356916,"statfs":{"total":773094113280,"available":772461481984,"internally_reserved":0,"allocated":45785088,"data_stored":23835494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":585092658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":565,"apply_latency_ms":565,"commit_latency_ns":565000000,"apply_latency_ns":565000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000717"},"pg_stats":[{"pgid":"11.1b","version":"0'0","reported_seq":1854,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.129142+0000","last_change":"2024-04-29T20:25:19.221101+0000","last_active":"2024-04-29T20:48:19.129142+0000","last_peered":"2024-04-29T20:48:19.129142+0000","last_clean":"2024-04-29T20:48:19.129142+0000","last_became_active":"2024-04-29T20:25:19.211321+0000","last_became_peered":"2024-04-29T20:25:19.211321+0000","last_unstale":"2024-04-29T20:48:19.129142+0000","last_undegraded":"2024-04-29T20:48:19.129142+0000","last_fullsized":"2024-04-29T20:48:19.129142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:42:34.869149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1846,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.552440+0000","last_change":"2024-04-29T20:25:24.951540+0000","last_active":"2024-04-29T20:48:19.552440+0000","last_peered":"2024-04-29T20:48:19.552440+0000","last_clean":"2024-04-29T20:48:19.552440+0000","last_became_active":"2024-04-29T20:25:24.924922+0000","last_became_peered":"2024-04-29T20:25:24.924922+0000","last_unstale":"2024-04-29T20:48:19.552440+0000","last_undegraded":"2024-04-29T20:48:19.552440+0000","last_fullsized":"2024-04-29T20:48:19.552440+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:01:14.186073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1847,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.548496+0000","last_change":"2024-04-29T20:25:25.197139+0000","last_active":"2024-04-29T20:48:19.548496+0000","last_peered":"2024-04-29T20:48:19.548496+0000","last_clean":"2024-04-29T20:48:19.548496+0000","last_became_active":"2024-04-29T20:25:25.191769+0000","last_became_peered":"2024-04-29T20:25:25.191769+0000","last_unstale":"2024-04-29T20:48:19.548496+0000","last_undegraded":"2024-04-29T20:48:19.548496+0000","last_fullsized":"2024-04-29T20:48:19.548496+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:57:41.743326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1853,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.515927+0000","last_change":"2024-04-29T20:25:26.047119+0000","last_active":"2024-04-29T20:48:18.515927+0000","last_peered":"2024-04-29T20:48:18.515927+0000","last_clean":"2024-04-29T20:48:18.515927+0000","last_became_active":"2024-04-29T20:25:26.041502+0000","last_became_peered":"2024-04-29T20:25:26.041502+0000","last_unstale":"2024-04-29T20:48:18.515927+0000","last_undegraded":"2024-04-29T20:48:18.515927+0000","last_fullsized":"2024-04-29T20:48:18.515927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:41:25.674969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1843,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-04-29T20:48:16.928208+0000","last_change":"2024-04-29T20:25:25.625395+0000","last_active":"2024-04-29T20:48:16.928208+0000","last_peered":"2024-04-29T20:48:16.928208+0000","last_clean":"2024-04-29T20:48:16.928208+0000","last_became_active":"2024-04-29T20:25:25.611019+0000","last_became_peered":"2024-04-29T20:25:25.611019+0000","last_unstale":"2024-04-29T20:48:16.928208+0000","last_undegraded":"2024-04-29T20:48:16.928208+0000","last_fullsized":"2024-04-29T20:48:16.928208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:57.199660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1864,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.925739+0000","last_change":"2024-04-29T20:25:22.691865+0000","last_active":"2024-04-29T20:48:17.925739+0000","last_peered":"2024-04-29T20:48:17.925739+0000","last_clean":"2024-04-29T20:48:17.925739+0000","last_became_active":"2024-04-29T20:25:22.685992+0000","last_became_peered":"2024-04-29T20:25:22.685992+0000","last_unstale":"2024-04-29T20:48:17.925739+0000","last_undegraded":"2024-04-29T20:48:17.925739+0000","last_fullsized":"2024-04-29T20:48:17.925739+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:23:46.806032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1859,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.196055+0000","last_change":"2024-04-29T20:25:26.419912+0000","last_active":"2024-04-29T20:48:19.196055+0000","last_peered":"2024-04-29T20:48:19.196055+0000","last_clean":"2024-04-29T20:48:19.196055+0000","last_became_active":"2024-04-29T20:25:26.414222+0000","last_became_peered":"2024-04-29T20:25:26.414222+0000","last_unstale":"2024-04-29T20:48:19.196055+0000","last_undegraded":"2024-04-29T20:48:19.196055+0000","last_fullsized":"2024-04-29T20:48:19.196055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:26:46.324672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1858,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.083584+0000","last_change":"2024-04-29T20:25:31.138205+0000","last_active":"2024-04-29T20:48:21.083584+0000","last_peered":"2024-04-29T20:48:21.083584+0000","last_clean":"2024-04-29T20:48:21.083584+0000","last_became_active":"2024-04-29T20:25:31.114951+0000","last_became_peered":"2024-04-29T20:25:31.114951+0000","last_unstale":"2024-04-29T20:48:21.083584+0000","last_undegraded":"2024-04-29T20:48:21.083584+0000","last_fullsized":"2024-04-29T20:48:21.083584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:57:40.889307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1859,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.557315+0000","last_change":"2024-04-29T20:25:27.559546+0000","last_active":"2024-04-29T20:48:19.557315+0000","last_peered":"2024-04-29T20:48:19.557315+0000","last_clean":"2024-04-29T20:48:19.557315+0000","last_became_active":"2024-04-29T20:25:27.553478+0000","last_became_peered":"2024-04-29T20:25:27.553478+0000","last_unstale":"2024-04-29T20:48:19.557315+0000","last_undegraded":"2024-04-29T20:48:19.557315+0000","last_fullsized":"2024-04-29T20:48:19.557315+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:36:09.125537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1866,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.794967+0000","last_change":"2024-04-29T20:25:06.859912+0000","last_active":"2024-04-29T20:48:17.794967+0000","last_peered":"2024-04-29T20:48:17.794967+0000","last_clean":"2024-04-29T20:48:17.794967+0000","last_became_active":"2024-04-29T20:25:06.850044+0000","last_became_peered":"2024-04-29T20:25:06.850044+0000","last_unstale":"2024-04-29T20:48:17.794967+0000","last_undegraded":"2024-04-29T20:48:17.794967+0000","last_fullsized":"2024-04-29T20:48:17.794967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:27:20.439378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"43'7","reported_seq":1883,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.073821+0000","last_change":"2024-04-29T20:25:09.449508+0000","last_active":"2024-04-29T20:48:21.073821+0000","last_peered":"2024-04-29T20:48:21.073821+0000","last_clean":"2024-04-29T20:48:21.073821+0000","last_became_active":"2024-04-29T20:25:09.413652+0000","last_became_peered":"2024-04-29T20:25:09.413652+0000","last_unstale":"2024-04-29T20:48:21.073821+0000","last_undegraded":"2024-04-29T20:48:21.073821+0000","last_fullsized":"2024-04-29T20:48:21.073821+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:51:59.474941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1858,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.519481+0000","last_change":"2024-04-29T20:25:21.119321+0000","last_active":"2024-04-29T20:48:18.519481+0000","last_peered":"2024-04-29T20:48:18.519481+0000","last_clean":"2024-04-29T20:48:18.519481+0000","last_became_active":"2024-04-29T20:25:21.109982+0000","last_became_peered":"2024-04-29T20:25:21.109982+0000","last_unstale":"2024-04-29T20:48:18.519481+0000","last_undegraded":"2024-04-29T20:48:18.519481+0000","last_fullsized":"2024-04-29T20:48:18.519481+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:58:45.347986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.793026+0000","last_change":"2024-04-29T20:25:04.146851+0000","last_active":"2024-04-29T20:48:17.793026+0000","last_peered":"2024-04-29T20:48:17.793026+0000","last_clean":"2024-04-29T20:48:17.793026+0000","last_became_active":"2024-04-29T20:25:04.139931+0000","last_became_peered":"2024-04-29T20:25:04.139931+0000","last_unstale":"2024-04-29T20:48:17.793026+0000","last_undegraded":"2024-04-29T20:48:17.793026+0000","last_fullsized":"2024-04-29T20:48:17.793026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:07:25.721846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":1881,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.132629+0000","last_change":"2024-04-29T20:22:57.765613+0000","last_active":"2024-04-29T20:48:19.132629+0000","last_peered":"2024-04-29T20:48:19.132629+0000","last_clean":"2024-04-29T20:48:19.132629+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:48:19.132629+0000","last_undegraded":"2024-04-29T20:48:19.132629+0000","last_fullsized":"2024-04-29T20:48:19.132629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:42.263517+0000","last_clean_scrub_stamp":"2024-04-29T20:22:42.263517+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:06:48.686106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.560572+0000","last_change":"2024-04-29T20:22:58.874338+0000","last_active":"2024-04-29T20:48:19.560572+0000","last_peered":"2024-04-29T20:48:19.560572+0000","last_clean":"2024-04-29T20:48:19.560572+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:48:19.560572+0000","last_undegraded":"2024-04-29T20:48:19.560572+0000","last_fullsized":"2024-04-29T20:48:19.560572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:59:51.304097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1858,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.546550+0000","last_change":"2024-04-29T20:25:07.978420+0000","last_active":"2024-04-29T20:48:19.546550+0000","last_peered":"2024-04-29T20:48:19.546550+0000","last_clean":"2024-04-29T20:48:19.546550+0000","last_became_active":"2024-04-29T20:25:07.961102+0000","last_became_peered":"2024-04-29T20:25:07.961102+0000","last_unstale":"2024-04-29T20:48:19.546550+0000","last_undegraded":"2024-04-29T20:48:19.546550+0000","last_fullsized":"2024-04-29T20:48:19.546550+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T00:27:36.921570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.539733+0000","last_change":"2024-04-29T20:22:58.155879+0000","last_active":"2024-04-29T20:48:19.539733+0000","last_peered":"2024-04-29T20:48:19.539733+0000","last_clean":"2024-04-29T20:48:19.539733+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:48:19.539733+0000","last_undegraded":"2024-04-29T20:48:19.539733+0000","last_fullsized":"2024-04-29T20:48:19.539733+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:40:44.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1862,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.517746+0000","last_change":"2024-04-29T20:25:16.331754+0000","last_active":"2024-04-29T20:48:18.517746+0000","last_peered":"2024-04-29T20:48:18.517746+0000","last_clean":"2024-04-29T20:48:18.517746+0000","last_became_active":"2024-04-29T20:25:16.326218+0000","last_became_peered":"2024-04-29T20:25:16.326218+0000","last_unstale":"2024-04-29T20:48:18.517746+0000","last_undegraded":"2024-04-29T20:48:18.517746+0000","last_fullsized":"2024-04-29T20:48:18.517746+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:34:14.827307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.721476+0000","last_change":"2024-04-29T20:22:58.100523+0000","last_active":"2024-04-29T20:48:17.721476+0000","last_peered":"2024-04-29T20:48:17.721476+0000","last_clean":"2024-04-29T20:48:17.721476+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:48:17.721476+0000","last_undegraded":"2024-04-29T20:48:17.721476+0000","last_fullsized":"2024-04-29T20:48:17.721476+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:31:35.619213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1864,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:18.532898+0000","last_change":"2024-04-29T20:25:06.176161+0000","last_active":"2024-04-29T20:48:18.532898+0000","last_peered":"2024-04-29T20:48:18.532898+0000","last_clean":"2024-04-29T20:48:18.532898+0000","last_became_active":"2024-04-29T20:25:06.170624+0000","last_became_peered":"2024-04-29T20:25:06.170624+0000","last_unstale":"2024-04-29T20:48:18.532898+0000","last_undegraded":"2024-04-29T20:48:18.532898+0000","last_fullsized":"2024-04-29T20:48:18.532898+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:38:50.008456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.549301+0000","last_change":"2024-04-29T20:22:58.456344+0000","last_active":"2024-04-29T20:48:19.549301+0000","last_peered":"2024-04-29T20:48:19.549301+0000","last_clean":"2024-04-29T20:48:19.549301+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:48:19.549301+0000","last_undegraded":"2024-04-29T20:48:19.549301+0000","last_fullsized":"2024-04-29T20:48:19.549301+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:02:31.378764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1859,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.792356+0000","last_change":"2024-04-29T20:25:19.733222+0000","last_active":"2024-04-29T20:48:17.792356+0000","last_peered":"2024-04-29T20:48:17.792356+0000","last_clean":"2024-04-29T20:48:17.792356+0000","last_became_active":"2024-04-29T20:25:19.725027+0000","last_became_peered":"2024-04-29T20:25:19.725027+0000","last_unstale":"2024-04-29T20:48:17.792356+0000","last_undegraded":"2024-04-29T20:48:17.792356+0000","last_fullsized":"2024-04-29T20:48:17.792356+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:51:22.081409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.860416+0000","last_change":"2024-04-29T20:22:58.466585+0000","last_active":"2024-04-29T20:48:17.860416+0000","last_peered":"2024-04-29T20:48:17.860416+0000","last_clean":"2024-04-29T20:48:17.860416+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:48:17.860416+0000","last_undegraded":"2024-04-29T20:48:17.860416+0000","last_fullsized":"2024-04-29T20:48:17.860416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:05:28.648611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.764121+0000","last_change":"2024-04-29T20:25:04.688244+0000","last_active":"2024-04-29T20:48:17.764121+0000","last_peered":"2024-04-29T20:48:17.764121+0000","last_clean":"2024-04-29T20:48:17.764121+0000","last_became_active":"2024-04-29T20:25:04.678852+0000","last_became_peered":"2024-04-29T20:25:04.678852+0000","last_unstale":"2024-04-29T20:48:17.764121+0000","last_undegraded":"2024-04-29T20:48:17.764121+0000","last_fullsized":"2024-04-29T20:48:17.764121+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:05:13.179776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.570994+0000","last_change":"2024-04-29T20:22:58.225857+0000","last_active":"2024-04-29T20:48:19.570994+0000","last_peered":"2024-04-29T20:48:19.570994+0000","last_clean":"2024-04-29T20:48:19.570994+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:48:19.570994+0000","last_undegraded":"2024-04-29T20:48:19.570994+0000","last_fullsized":"2024-04-29T20:48:19.570994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:16:31.016297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1868,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.103740+0000","last_change":"2024-04-29T20:25:05.054910+0000","last_active":"2024-04-29T20:48:21.103740+0000","last_peered":"2024-04-29T20:48:21.103740+0000","last_clean":"2024-04-29T20:48:21.103740+0000","last_became_active":"2024-04-29T20:25:05.045385+0000","last_became_peered":"2024-04-29T20:25:05.045385+0000","last_unstale":"2024-04-29T20:48:21.103740+0000","last_undegraded":"2024-04-29T20:48:21.103740+0000","last_fullsized":"2024-04-29T20:48:21.103740+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:53:43.926594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1872,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.568074+0000","last_change":"2024-04-29T20:22:59.053068+0000","last_active":"2024-04-29T20:48:19.568074+0000","last_peered":"2024-04-29T20:48:19.568074+0000","last_clean":"2024-04-29T20:48:19.568074+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:48:19.568074+0000","last_undegraded":"2024-04-29T20:48:19.568074+0000","last_fullsized":"2024-04-29T20:48:19.568074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:40:41.996522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1868,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.086257+0000","last_change":"2024-04-29T20:25:19.272741+0000","last_active":"2024-04-29T20:48:21.086257+0000","last_peered":"2024-04-29T20:48:21.086257+0000","last_clean":"2024-04-29T20:48:21.086257+0000","last_became_active":"2024-04-29T20:25:19.206413+0000","last_became_peered":"2024-04-29T20:25:19.206413+0000","last_unstale":"2024-04-29T20:48:21.086257+0000","last_undegraded":"2024-04-29T20:48:21.086257+0000","last_fullsized":"2024-04-29T20:48:21.086257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:16:10.608426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1869,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.592275+0000","last_change":"2024-04-29T20:22:59.008821+0000","last_active":"2024-04-29T20:48:19.592275+0000","last_peered":"2024-04-29T20:48:19.592275+0000","last_clean":"2024-04-29T20:48:19.592275+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:48:19.592275+0000","last_undegraded":"2024-04-29T20:48:19.592275+0000","last_fullsized":"2024-04-29T20:48:19.592275+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:05:01.384817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1853,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.143095+0000","last_change":"2024-04-29T20:25:19.599585+0000","last_active":"2024-04-29T20:48:19.143095+0000","last_peered":"2024-04-29T20:48:19.143095+0000","last_clean":"2024-04-29T20:48:19.143095+0000","last_became_active":"2024-04-29T20:25:19.589991+0000","last_became_peered":"2024-04-29T20:25:19.589991+0000","last_unstale":"2024-04-29T20:48:19.143095+0000","last_undegraded":"2024-04-29T20:48:19.143095+0000","last_fullsized":"2024-04-29T20:48:19.143095+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:46:47.268065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1845,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.192864+0000","last_change":"2024-04-29T20:25:24.677178+0000","last_active":"2024-04-29T20:48:19.192864+0000","last_peered":"2024-04-29T20:48:19.192864+0000","last_clean":"2024-04-29T20:48:19.192864+0000","last_became_active":"2024-04-29T20:25:24.662027+0000","last_became_peered":"2024-04-29T20:25:24.662027+0000","last_unstale":"2024-04-29T20:48:19.192864+0000","last_undegraded":"2024-04-29T20:48:19.192864+0000","last_fullsized":"2024-04-29T20:48:19.192864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:37:00.015485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1858,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.583482+0000","last_change":"2024-04-29T20:25:26.584778+0000","last_active":"2024-04-29T20:48:19.583482+0000","last_peered":"2024-04-29T20:48:19.583482+0000","last_clean":"2024-04-29T20:48:19.583482+0000","last_became_active":"2024-04-29T20:25:26.576214+0000","last_became_peered":"2024-04-29T20:25:26.576214+0000","last_unstale":"2024-04-29T20:48:19.583482+0000","last_undegraded":"2024-04-29T20:48:19.583482+0000","last_fullsized":"2024-04-29T20:48:19.583482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:20:55.927114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.840203+0000","last_change":"2024-04-29T20:25:02.740652+0000","last_active":"2024-04-29T20:48:17.840203+0000","last_peered":"2024-04-29T20:48:17.840203+0000","last_clean":"2024-04-29T20:48:17.840203+0000","last_became_active":"2024-04-29T20:25:02.734276+0000","last_became_peered":"2024-04-29T20:25:02.734276+0000","last_unstale":"2024-04-29T20:48:17.840203+0000","last_undegraded":"2024-04-29T20:48:17.840203+0000","last_fullsized":"2024-04-29T20:48:17.840203+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:34.054845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1865,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.929618+0000","last_change":"2024-04-29T20:25:06.026305+0000","last_active":"2024-04-29T20:48:17.929618+0000","last_peered":"2024-04-29T20:48:17.929618+0000","last_clean":"2024-04-29T20:48:17.929618+0000","last_became_active":"2024-04-29T20:25:06.011110+0000","last_became_peered":"2024-04-29T20:25:06.011110+0000","last_unstale":"2024-04-29T20:48:17.929618+0000","last_undegraded":"2024-04-29T20:48:17.929618+0000","last_fullsized":"2024-04-29T20:48:17.929618+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:51:47.148992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1866,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.720492+0000","last_change":"2024-04-29T20:25:02.267425+0000","last_active":"2024-04-29T20:48:17.720492+0000","last_peered":"2024-04-29T20:48:17.720492+0000","last_clean":"2024-04-29T20:48:17.720492+0000","last_became_active":"2024-04-29T20:25:02.256407+0000","last_became_peered":"2024-04-29T20:25:02.256407+0000","last_unstale":"2024-04-29T20:48:17.720492+0000","last_undegraded":"2024-04-29T20:48:17.720492+0000","last_fullsized":"2024-04-29T20:48:17.720492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:42.168617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1857,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.134309+0000","last_change":"2024-04-29T20:25:05.528506+0000","last_active":"2024-04-29T20:48:19.134309+0000","last_peered":"2024-04-29T20:48:19.134309+0000","last_clean":"2024-04-29T20:48:19.134309+0000","last_became_active":"2024-04-29T20:25:05.517205+0000","last_became_peered":"2024-04-29T20:25:05.517205+0000","last_unstale":"2024-04-29T20:48:19.134309+0000","last_undegraded":"2024-04-29T20:48:19.134309+0000","last_fullsized":"2024-04-29T20:48:19.134309+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:03:11.364243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1858,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.182427+0000","last_change":"2024-04-29T20:25:11.189601+0000","last_active":"2024-04-29T20:48:19.182427+0000","last_peered":"2024-04-29T20:48:19.182427+0000","last_clean":"2024-04-29T20:48:19.182427+0000","last_became_active":"2024-04-29T20:25:11.174336+0000","last_became_peered":"2024-04-29T20:25:11.174336+0000","last_unstale":"2024-04-29T20:48:19.182427+0000","last_undegraded":"2024-04-29T20:48:19.182427+0000","last_fullsized":"2024-04-29T20:48:19.182427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:12:54.605306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1857,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.578483+0000","last_change":"2024-04-29T20:25:09.447149+0000","last_active":"2024-04-29T20:48:19.578483+0000","last_peered":"2024-04-29T20:48:19.578483+0000","last_clean":"2024-04-29T20:48:19.578483+0000","last_became_active":"2024-04-29T20:25:09.429696+0000","last_became_peered":"2024-04-29T20:25:09.429696+0000","last_unstale":"2024-04-29T20:48:19.578483+0000","last_undegraded":"2024-04-29T20:48:19.578483+0000","last_fullsized":"2024-04-29T20:48:19.578483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:40.988418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1863,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:17.889835+0000","last_change":"2024-04-29T20:25:22.557015+0000","last_active":"2024-04-29T20:48:17.889835+0000","last_peered":"2024-04-29T20:48:17.889835+0000","last_clean":"2024-04-29T20:48:17.889835+0000","last_became_active":"2024-04-29T20:25:22.551405+0000","last_became_peered":"2024-04-29T20:25:22.551405+0000","last_unstale":"2024-04-29T20:48:17.889835+0000","last_undegraded":"2024-04-29T20:48:17.889835+0000","last_fullsized":"2024-04-29T20:48:17.889835+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:48:10.829532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1856,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-04-29T20:48:19.188229+0000","last_change":"2024-04-29T20:25:12.278620+0000","last_active":"2024-04-29T20:48:19.188229+0000","last_peered":"2024-04-29T20:48:19.188229+0000","last_clean":"2024-04-29T20:48:19.188229+0000","last_became_active":"2024-04-29T20:25:12.263826+0000","last_became_peered":"2024-04-29T20:25:12.263826+0000","last_unstale":"2024-04-29T20:48:19.188229+0000","last_undegraded":"2024-04-29T20:48:19.188229+0000","last_fullsized":"2024-04-29T20:48:19.188229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:32:24.000890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1867,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T20:48:21.101672+0000","last_change":"2024-04-29T20:25:09.882543+0000","last_active":"2024-04-29T20:48:21.101672+0000","last_peered":"2024-04-29T20:48:21.101672+0000","last_clean":"2024-04-29T20:48:21.101672+0000","last_became_active":"2024-04-29T20:25:09.849524+0000","last_became_peered":"2024-04-29T20:25:09.849524+0000","last_unstale":"2024-04-29T20:48:21.101672+0000","last_undegraded":"2024-04-29T20:48:21.101672+0000","last_fullsized":"2024-04-29T20:48:21.101672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:32:59.196210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280821,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81080,"kb_used_data":5452,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290760,"statfs":{"total":96636764160,"available":96553738240,"internally_reserved":0,"allocated":5582848,"data_stored":2832718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":98,"apply_latency_ms":98,"commit_latency_ns":98000000,"apply_latency_ns":98000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280821,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81904,"kb_used_data":5944,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289936,"statfs":{"total":96636764160,"available":96552894464,"internally_reserved":0,"allocated":6086656,"data_stored":3342363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280820,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68604,"kb_used_data":5456,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303236,"statfs":{"total":96636764160,"available":96566513664,"internally_reserved":0,"allocated":5586944,"data_stored":2832723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":166,"apply_latency_ms":166,"commit_latency_ns":166000000,"apply_latency_ns":166000000},"alerts":[]},{"osd":6,"up_from":20,"seq":85899346231,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81604,"kb_used_data":5460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290236,"statfs":{"total":96636764160,"available":96553201664,"internally_reserved":0,"allocated":5591040,"data_stored":2854406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509748,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81296,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290544,"statfs":{"total":96636764160,"available":96553517056,"internally_reserved":0,"allocated":5623808,"data_stored":2887973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509749,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69036,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302804,"statfs":{"total":96636764160,"available":96566071296,"internally_reserved":0,"allocated":6033408,"data_stored":3294005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509750,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73244,"kb_used_data":5576,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298596,"statfs":{"total":96636764160,"available":96561762304,"internally_reserved":0,"allocated":5709824,"data_stored":2965285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81036,"kb_used_data":5440,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290804,"statfs":{"total":96636764160,"available":96553783296,"internally_reserved":0,"allocated":5570560,"data_stored":2826021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:48:22.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-29T20:48:22.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.983+0000 7f5d04759640 10 monclient: get_monmap_and_config 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.983+0000 7f5d04759640 10 monclient: build_initial_monmap 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.983+0000 7f5d04759640 10 monclient: monmap: 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:22.984416+0000 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:22.984416+0000 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:22.984 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:22.985 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 Processor -- start 2024-04-29T20:48:22.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 -- start start 2024-04-29T20:48:22.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient: init 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5cfc067990 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc0b0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): picked mon.noname-b con 0x7f5cfc0b1280 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f5cfc140420 0x7f5cfc1408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): picked mon.noname-c con 0x7f5cfc140420 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cfc0b2550 con 0x7f5cfc067990 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f5cfc0b2320 con 0x7f5cfc140420 2024-04-29T20:48:22.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5cfc140df0 con 0x7f5cfc0b1280 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.984+0000 7f5d04759640 10 monclient(hunting): authenticate will time out at 3052.144546s 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d01cf4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc0b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d024f5640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5cfc067990 0x7f5cfc146950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59492/0 (socket says 172.21.15.64:59492) 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d024f5640 1 -- 172.21.15.64:0/1582240219 learned_addr learned my addr 172.21.15.64:0/1582240219 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d02cf6640 1 --2- 172.21.15.64:0/1582240219 >> v2:172.21.15.167:3300/0 conn(0x7f5cfc140420 0x7f5cfc1408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d02cf6640 10 monclient(hunting): get_auth_request con 0x7f5cfc140420 auth_method 0 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d02cf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d02cf6640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.985+0000 7f5d02cf6640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): get_auth_request con 0x7f5cfc0b1280 auth_method 0 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.986+0000 7f5d01cf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_done global_id 22129 payload 274 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 1 -- 172.21.15.64:0/1582240219 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 msgr2=0x7f5cfc0b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 1 --2- 172.21.15.64:0/1582240219 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc0b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 1 -- 172.21.15.64:0/1582240219 >> v1:172.21.15.64:6789/0 conn(0x7f5cfc067990 legacy=0x7f5cfc146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 10 monclient: _finish_hunting 0 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 1 monclient: found mon.noname-c 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:22.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.988+0000 7f5d02cf6640 1 -- 172.21.15.64:0/1582240219 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc0b0d60 con 0x7f5cfc140420 2024-04-29T20:48:22.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.990+0000 7f5d02cf6640 1 --2- 172.21.15.64:0/1582240219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc140420 0x7f5cfc1408b0 secure :-1 s=READY pgs=5618 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009d90 tx=0x7f5cf402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f6ad83f41b091d server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:22.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 1 -- 172.21.15.64:0/1582240219 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4039070 con 0x7f5cfc140420 2024-04-29T20:48:22.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:22.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:22.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: dump: 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: _finish_auth 0 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: _check_auth_tickets 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 1 -- 172.21.15.64:0/1582240219 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf402cbc0 con 0x7f5cfc140420 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 1 -- 172.21.15.64:0/1582240219 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4034440 con 0x7f5cfc140420 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d014f3640 10 monclient: dump: 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:22.995 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d04759640 5 monclient: authenticate success, global_id 22129 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d04759640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:22.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.993+0000 7f5d04759640 4 set_mon_vals no callback set 2024-04-29T20:48:22.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 10 monclient: shutdown 2024-04-29T20:48:22.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 -- 172.21.15.64:0/1582240219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc140420 msgr2=0x7f5cfc1408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 --2- 172.21.15.64:0/1582240219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc140420 0x7f5cfc1408b0 secure :-1 s=READY pgs=5618 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009d90 tx=0x7f5cf402be10 comp rx=0 tx=0).stop 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 -- 172.21.15.64:0/1582240219 shutdown_connections 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 --2- 172.21.15.64:0/1582240219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc140420 0x7f5cfc1408b0 unknown :-1 s=CLOSED pgs=5618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 --2- 172.21.15.64:0/1582240219 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc0b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 -- 172.21.15.64:0/1582240219 >> 172.21.15.64:0/1582240219 conn(0x7f5cfc0b5d80 msgr2=0x7f5cfc0b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 -- 172.21.15.64:0/1582240219 shutdown_connections 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.994+0000 7f5d04759640 1 -- 172.21.15.64:0/1582240219 wait complete. 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 10 monclient: build_initial_monmap 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 10 monclient: monmap: 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 1 librados: starting msgr at 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 1 librados: starting objecter 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 1 Processor -- start 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 1 -- start start 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 1 librados: setting wanted keys 2024-04-29T20:48:23.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 1 librados: calling monclient init 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.995+0000 7f5d04759640 10 monclient: init 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc067990 0x7f5cfc1dc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): picked mon.noname-b con 0x7f5cfc067990 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc13bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): picked mon.noname-c con 0x7f5cfc0b1280 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cfc140420 0x7f5cfc13c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): picked mon.noname-a con 0x7f5cfc140420 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1471e0 con 0x7f5cfc140420 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc147060 con 0x7f5cfc067990 2024-04-29T20:48:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5cfc14dce0 con 0x7f5cfc0b1280 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d04759640 10 monclient(hunting): authenticate will time out at 3052.155979s 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.996+0000 7f5d01cf4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc13bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cfc140420 0x7f5cfc13c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc067990 0x7f5cfc1dc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc13bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:58302/0 (socket says 172.21.15.64:58302) 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 1 -- 172.21.15.64:0/286377904 learned_addr learned my addr 172.21.15.64:0/286377904 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): get_auth_request con 0x7f5cfc0b1280 auth_method 0 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:23.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): get_auth_request con 0x7f5cfc067990 auth_method 0 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): get_auth_request con 0x7f5cfc140420 auth_method 0 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d024f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d01cf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:22.997+0000 7f5d02cf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 10 monclient(hunting): handle_auth_done global_id 22132 payload 824 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 1 -- 172.21.15.64:0/286377904 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 msgr2=0x7f5cfc13bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc13bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 1 -- 172.21.15.64:0/286377904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cfc140420 msgr2=0x7f5cfc13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cfc140420 0x7f5cfc13c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 10 monclient: _finish_hunting 0 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 1 monclient: found mon.noname-b 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:23.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.000+0000 7f5d024f5640 1 -- 172.21.15.64:0/286377904 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc146560 con 0x7f5cfc067990 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.002+0000 7f5d024f5640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc067990 0x7f5cfc1dc110 secure :-1 s=READY pgs=5619 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c370 tx=0x7f5cf800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.002+0000 7f5cf2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cfc067990 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.002+0000 7f5cf2ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8017070 con 0x7f5cfc067990 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: dump: 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:23.007 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: _finish_auth 0 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8005ce0 con 0x7f5cfc067990 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8007310 con 0x7f5cfc067990 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5cf2ffd640 10 monclient: dump: 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:23.008 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 5 monclient: authenticate success, global_id 22132 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 10 monclient: _renew_subs 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d00cf2640 4 set_mon_vals no callback set 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cfc146840 con 0x7f5cfc067990 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 10 client.22132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:23.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 10 monclient: _renew_subs 2024-04-29T20:48:23.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:23.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cfc145ae0 con 0x7f5cfc067990 2024-04-29T20:48:23.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.006+0000 7f5d04759640 1 librados: init done 2024-04-29T20:48:23.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.007+0000 7f5d04759640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:23.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.007+0000 7f5d04759640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:23.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.007+0000 7f5d04759640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:23.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.007+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cfc1dcb50 con 0x7f5cfc067990 2024-04-29T20:48:23.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.261+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5cf801a070 con 0x7f5cfc067990 2024-04-29T20:48:23.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.261+0000 7f5cf2ffd640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5ce006e2d0 0x7f5ce0070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:23.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.261+0000 7f5d01cf4640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5ce006e2d0 0x7f5ce0070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:23.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.261+0000 7f5d01cf4640 10 monclient: get_auth_request con 0x7f5ce006e2d0 auth_method 0 2024-04-29T20:48:23.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.262+0000 7f5cf2ffd640 10 client.22132.objecter ms_handle_connect 0x7f5ce006e2d0 2024-04-29T20:48:23.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.262+0000 7f5d01cf4640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5ce006e2d0 0x7f5ce0070790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0b30e0 tx=0x7f5cec009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:23.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.264+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(975..975 src has 638..975) ==== 6337+0+0 (secure 0 0 0) 0x7f5cf80570c0 con 0x7f5cfc067990 2024-04-29T20:48:23.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.264+0000 7f5cf2ffd640 10 client.22132.objecter ms_dispatch 0x7f5cfc000b90 osd_map(975..975 src has 638..975) 2024-04-29T20:48:23.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.264+0000 7f5cf2ffd640 3 client.22132.objecter handle_osd_map got epochs [975,975] > 0 2024-04-29T20:48:23.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.264+0000 7f5cf2ffd640 3 client.22132.objecter handle_osd_map decoding full epoch 975 2024-04-29T20:48:23.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.264+0000 7f5cf2ffd640 20 client.22132.objecter dump_active .. 0 homeless 2024-04-29T20:48:23.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.278+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5cf8021020 con 0x7f5cfc067990 2024-04-29T20:48:23.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.278+0000 7f5cf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:23.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.278+0000 7f5cf2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:23.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.468+0000 7f5d04759640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:48:23.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.468+0000 7f5d04759640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:48:23.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.468+0000 7f5d04759640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:23.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.468+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5cfc04fce0 con 0x7f5cfc067990 2024-04-29T20:48:23.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.504+0000 7f5cf2ffd640 1 -- 172.21.15.64:0/286377904 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v975) ==== 74+0+17726 (secure 0 0 0) 0x7f5cfc145ae0 con 0x7f5cfc067990 2024-04-29T20:48:23.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.504+0000 7f5cf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-29T20:48:23.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.504+0000 7f5cf2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:48:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:48:23.505 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":975,"fsid":"2216f09b-4679-4a15-bb1e-fe6d8ebb49fc","created":"2024-04-29T20:21:53.494036+0000","modified":"2024-04-29T20:48:20.404253+0000","last_up_change":"2024-04-29T20:22:54.303806+0000","last_in_change":"2024-04-29T20:21:59.194193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":104,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T20:22:41.558205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-29T20:22:56.229418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-29T20:24:51.670798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ab5e452-c2c0-4817-81db-f5fc686af04c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6810","nonce":2796311601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6815","nonce":2796311601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6823","nonce":2796311601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2796311601},{"type":"v1","addr":"172.21.15.64:6821","nonce":2796311601}]},"public_addr":"172.21.15.64:6810/2796311601","cluster_addr":"172.21.15.64:6815/2796311601","heartbeat_back_addr":"172.21.15.64:6823/2796311601","heartbeat_front_addr":"172.21.15.64:6821/2796311601","state":["exists","up"]},{"osd":1,"uuid":"c736ff85-13c2-4c71-b5e3-b603c9023744","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6811","nonce":3421706551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6814","nonce":3421706551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6820","nonce":3421706551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3421706551},{"type":"v1","addr":"172.21.15.64:6817","nonce":3421706551}]},"public_addr":"172.21.15.64:6811/3421706551","cluster_addr":"172.21.15.64:6814/3421706551","heartbeat_back_addr":"172.21.15.64:6820/3421706551","heartbeat_front_addr":"172.21.15.64:6817/3421706551","state":["exists","up"]},{"osd":2,"uuid":"2d380b76-312c-43ca-8301-7ec705d9fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6801","nonce":2373308255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6803","nonce":2373308255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6807","nonce":2373308255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2373308255},{"type":"v1","addr":"172.21.15.64:6805","nonce":2373308255}]},"public_addr":"172.21.15.64:6801/2373308255","cluster_addr":"172.21.15.64:6803/2373308255","heartbeat_back_addr":"172.21.15.64:6807/2373308255","heartbeat_front_addr":"172.21.15.64:6805/2373308255","state":["exists","up"]},{"osd":3,"uuid":"2190b44c-dc16-4b94-9a4c-931c87d0e875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6825","nonce":471606907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6827","nonce":471606907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6831","nonce":471606907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":471606907},{"type":"v1","addr":"172.21.15.64:6829","nonce":471606907}]},"public_addr":"172.21.15.64:6825/471606907","cluster_addr":"172.21.15.64:6827/471606907","heartbeat_back_addr":"172.21.15.64:6831/471606907","heartbeat_front_addr":"172.21.15.64:6829/471606907","state":["exists","up"]},{"osd":4,"uuid":"f20c5825-9198-44fb-9b31-7de381a3310c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6827","nonce":3754201859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6829","nonce":3754201859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6833","nonce":3754201859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":3754201859},{"type":"v1","addr":"172.21.15.167:6831","nonce":3754201859}]},"public_addr":"172.21.15.167:6827/3754201859","cluster_addr":"172.21.15.167:6829/3754201859","heartbeat_back_addr":"172.21.15.167:6833/3754201859","heartbeat_front_addr":"172.21.15.167:6831/3754201859","state":["exists","up"]},{"osd":5,"uuid":"a4937fcb-dbd8-4cc1-ac32-ddf147f0469b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6803","nonce":3447508053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6805","nonce":3447508053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6809","nonce":3447508053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":3447508053},{"type":"v1","addr":"172.21.15.167:6807","nonce":3447508053}]},"public_addr":"172.21.15.167:6803/3447508053","cluster_addr":"172.21.15.167:6805/3447508053","heartbeat_back_addr":"172.21.15.167:6809/3447508053","heartbeat_front_addr":"172.21.15.167:6807/3447508053","state":["exists","up"]},{"osd":6,"uuid":"ac61b59e-81b4-47cd-a33c-aa3d02142b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6819","nonce":3916781155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6821","nonce":3916781155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6825","nonce":3916781155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":3916781155},{"type":"v1","addr":"172.21.15.167:6823","nonce":3916781155}]},"public_addr":"172.21.15.167:6819/3916781155","cluster_addr":"172.21.15.167:6821/3916781155","heartbeat_back_addr":"172.21.15.167:6825/3916781155","heartbeat_front_addr":"172.21.15.167:6823/3916781155","state":["exists","up"]},{"osd":7,"uuid":"f4d72d02-c58f-4ff9-8b55-f6de73ba6460","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6811","nonce":2258625350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6813","nonce":2258625350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6817","nonce":2258625350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":2258625350},{"type":"v1","addr":"172.21.15.167:6815","nonce":2258625350}]},"public_addr":"172.21.15.167:6811/2258625350","cluster_addr":"172.21.15.167:6813/2258625350","heartbeat_back_addr":"172.21.15.167:6817/2258625350","heartbeat_front_addr":"172.21.15.167:6815/2258625350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:40.154276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.540513+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:38.282652+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:39.442559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.557528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.720225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:35.815755+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T20:22:36.803337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":3,"length":2}]},{"pool":19,"snaps":[{"begin":1,"length":1}]},{"pool":26,"snaps":[{"begin":3,"length":4}]},{"pool":74,"snaps":[{"begin":1,"length":1}]},{"pool":114,"snaps":[{"begin":7,"length":2}]},{"pool":120,"snaps":[{"begin":1,"length":1}]},{"pool":133,"snaps":[{"begin":1,"length":2}]},{"pool":149,"snaps":[{"begin":3,"length":2}]},{"pool":151,"snaps":[{"begin":3,"length":1}]},{"pool":160,"snaps":[{"begin":2,"length":2}]},{"pool":165,"snaps":[{"begin":4,"length":2}]},{"pool":168,"snaps":[{"begin":2,"length":3}]},{"pool":178,"snaps":[{"begin":2,"length":2}]},{"pool":182,"snaps":[{"begin":2,"length":1}]},{"pool":186,"snaps":[{"begin":3,"length":2}]},{"pool":198,"snaps":[{"begin":5,"length":2}]},{"pool":201,"snaps":[{"begin":4,"length":4}]},{"pool":260,"snaps":[{"begin":7,"length":1}]},{"pool":270,"snaps":[{"begin":8,"length":1}]},{"pool":274,"snaps":[{"begin":10,"length":1}]},{"pool":276,"snaps":[{"begin":11,"length":2}]},{"pool":279,"snaps":[{"begin":14,"length":1}]},{"pool":322,"snaps":[{"begin":2,"length":1}]},{"pool":323,"snaps":[{"begin":3,"length":1}]},{"pool":326,"snaps":[{"begin":4,"length":1}]},{"pool":330,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T20:48:23.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 10 librados: watch_flush enter 2024-04-29T20:48:23.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 10 librados: watch_flush exit 2024-04-29T20:48:23.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 20 client.22132.objecter shutdown clearing up homeless session... 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 10 client.22132.objecter successfully canceled tick 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5ce006e2d0 msgr2=0x7f5ce0070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5ce006e2d0 0x7f5ce0070790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0b30e0 tx=0x7f5cec009190 comp rx=0 tx=0).stop 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 10 monclient: shutdown 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc067990 msgr2=0x7f5cfc1dc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc067990 0x7f5cfc1dc110 secure :-1 s=READY pgs=5619 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c370 tx=0x7f5cf800c830 comp rx=0 tx=0).stop 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 shutdown_connections 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5ce006e2d0 0x7f5ce0070790 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5cfc0b1280 0x7f5cfc13bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cfc140420 0x7f5cfc13c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 --2- 172.21.15.64:0/286377904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5cfc067990 0x7f5cfc1dc110 unknown :-1 s=CLOSED pgs=5619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 >> 172.21.15.64:0/286377904 conn(0x7f5cfc0b5d80 msgr2=0x7f5cfc0b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:23.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 shutdown_connections 2024-04-29T20:48:23.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 -- 172.21.15.64:0/286377904 wait complete. 2024-04-29T20:48:23.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:23.506+0000 7f5d04759640 1 librados: shutdown 2024-04-29T20:48:24.518 INFO:tasks.ceph:Scrubbing osd.0 2024-04-29T20:48:24.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:24.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.624+0000 7f1dbd08c640 10 monclient: get_monmap_and_config 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.624+0000 7f1dbd08c640 10 monclient: build_initial_monmap 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.624+0000 7f1dbd08c640 10 monclient: monmap: 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:24.624889+0000 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:24.624889+0000 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:24.624 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:24.625 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:24.625 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:24.625 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.624+0000 7f1dbd08c640 1 Processor -- start 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.624+0000 7f1dbd08c640 1 -- start start 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.624+0000 7f1dbd08c640 10 monclient: init 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f1db80b5b90 0x7f1db8146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): picked mon.noname-a con 0x7f1db80b5b90 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b5f70 0x7f1db80aa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): picked mon.noname-b con 0x7f1db80b5f70 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:24.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1db80b2d70 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db80aabc0 con 0x7f1db80b2d70 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f1db80aa9a0 con 0x7f1db80b5b90 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f1db80b2530 con 0x7f1db80b5f70 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1dbd08c640 10 monclient(hunting): authenticate will time out at 3053.784972s 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.625+0000 7f1db659c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b5f70 0x7f1db80aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db6d9d640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f1db80b5b90 0x7f1db8146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db759e640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f1db80b2d70 0x7f1db80b1e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:60146/0 (socket says 172.21.15.64:60146) 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db759e640 1 -- 172.21.15.64:0/2411176116 learned_addr learned my addr 172.21.15.64:0/2411176116 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db6d9d640 10 monclient(hunting): get_auth_request con 0x7f1db80b5b90 auth_method 0 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db659c640 10 monclient(hunting): get_auth_request con 0x7f1db80b5f70 auth_method 0 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db659c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.626+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.627+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.627+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.627+0000 7f1db659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.627+0000 7f1db659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.627+0000 7f1db659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.628+0000 7f1db5d9b640 1 -- 172.21.15.64:0/2411176116 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346251803 0 0) 0x7f1db80aabc0 con 0x7f1db80b2d70 2024-04-29T20:48:24.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.628+0000 7f1db5d9b640 10 monclient(hunting): _negotiate 2024-04-29T20:48:24.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.628+0000 7f1db5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.628+0000 7f1db5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.628+0000 7f1db5d9b640 10 monclient(hunting): my global_id is 22135 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.628+0000 7f1db5d9b640 1 -- 172.21.15.64:0/2411176116 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1da0003520 con 0x7f1db80b2d70 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 10 monclient(hunting): handle_auth_done global_id 16349 payload 274 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 1 -- 172.21.15.64:0/2411176116 >> v2:172.21.15.64:3300/0 conn(0x7f1db80b5b90 msgr2=0x7f1db8146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 1 --2- 172.21.15.64:0/2411176116 >> v2:172.21.15.64:3300/0 conn(0x7f1db80b5b90 0x7f1db8146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 1 -- 172.21.15.64:0/2411176116 >> v1:172.21.15.167:6789/0 conn(0x7f1db80b2d70 legacy=0x7f1db80b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 10 monclient: _finish_hunting 0 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 1 monclient: found mon.noname-b 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db659c640 1 -- 172.21.15.64:0/2411176116 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db80b26b0 con 0x7f1db80b5f70 2024-04-29T20:48:24.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.629+0000 7f1db6d9d640 1 --2- 172.21.15.64:0/2411176116 >> v2:172.21.15.64:3300/0 conn(0x7f1db80b5b90 0x7f1db8146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-29T20:48:24.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.631+0000 7f1db659c640 1 --2- 172.21.15.64:0/2411176116 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b5f70 0x7f1db80aa460 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0097a0 tx=0x7f1dac02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cfa41d37008ae33 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 1 -- 172.21.15.64:0/2411176116 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac035020 con 0x7f1db80b5f70 2024-04-29T20:48:24.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:24.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: dump: 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: _finish_auth 0 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: _check_auth_tickets 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 1 -- 172.21.15.64:0/2411176116 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dac031830 con 0x7f1db80b5f70 2024-04-29T20:48:24.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 1 -- 172.21.15.64:0/2411176116 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac031c00 con 0x7f1db80b5f70 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1dbd08c640 5 monclient: authenticate success, global_id 16349 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1dbd08c640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1dbd08c640 4 set_mon_vals no callback set 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.635+0000 7f1db5d9b640 10 monclient: dump: 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.637 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.638 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.638 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.638 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.638 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.638 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.638 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 10 monclient: shutdown 2024-04-29T20:48:24.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 -- 172.21.15.64:0/2411176116 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b5f70 msgr2=0x7f1db80aa460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 --2- 172.21.15.64:0/2411176116 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b5f70 0x7f1db80aa460 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0097a0 tx=0x7f1dac02b8d0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 -- 172.21.15.64:0/2411176116 shutdown_connections 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 --2- 172.21.15.64:0/2411176116 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b5f70 0x7f1db80aa460 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 --2- 172.21.15.64:0/2411176116 >> v2:172.21.15.64:3300/0 conn(0x7f1db80b5b90 0x7f1db8146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 -- 172.21.15.64:0/2411176116 >> 172.21.15.64:0/2411176116 conn(0x7f1db806c9d0 msgr2=0x7f1db806cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 -- 172.21.15.64:0/2411176116 shutdown_connections 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.636+0000 7f1dbd08c640 1 -- 172.21.15.64:0/2411176116 wait complete. 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.637+0000 7f1dbd08c640 10 monclient: build_initial_monmap 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.637+0000 7f1dbd08c640 10 monclient: monmap: 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:24.642 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.637+0000 7f1dbd08c640 1 librados: starting msgr at 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.637+0000 7f1dbd08c640 1 librados: starting objecter 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 Processor -- start 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 -- start start 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 librados: setting wanted keys 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 librados: calling monclient init 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient: init 2024-04-29T20:48:24.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b2d70 0x7f1db813ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): picked mon.noname-c con 0x7f1db80b2d70 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b5b90 0x7f1db81427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): picked mon.noname-b con 0x7f1db80b5b90 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1db8141200 0x7f1db8143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): picked mon.noname-a con 0x7f1db8141200 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1db80b55d0 con 0x7f1db8141200 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1db80b5450 con 0x7f1db80b5b90 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f1db814dc90 con 0x7f1db80b2d70 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.638+0000 7f1dbd08c640 10 monclient(hunting): authenticate will time out at 3053.798246s 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db759e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1db8141200 0x7f1db8143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b2d70 0x7f1db813ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db659c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b5b90 0x7f1db81427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b2d70 0x7f1db813ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45588/0 (socket says 172.21.15.64:45588) 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 1 -- 172.21.15.64:0/951540535 learned_addr learned my addr 172.21.15.64:0/951540535 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 10 monclient(hunting): get_auth_request con 0x7f1db80b2d70 auth_method 0 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db759e640 10 monclient(hunting): get_auth_request con 0x7f1db8141200 auth_method 0 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db759e640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db659c640 10 monclient(hunting): get_auth_request con 0x7f1db80b5b90 auth_method 0 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db659c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.639+0000 7f1db659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.640+0000 7f1db659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 10 monclient(hunting): handle_auth_done global_id 16352 payload 824 2024-04-29T20:48:24.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 1 -- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b5b90 msgr2=0x7f1db81427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b5b90 0x7f1db81427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 1 -- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1db8141200 msgr2=0x7f1db8143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1db8141200 0x7f1db8143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 10 monclient: _finish_hunting 0 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 1 monclient: found mon.noname-c 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.643+0000 7f1db6d9d640 1 -- 172.21.15.64:0/951540535 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db80ad280 con 0x7f1db80b2d70 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.645+0000 7f1db6d9d640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b2d70 0x7f1db813ed80 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009f60 tx=0x7f1db0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.645+0000 7f1d9f7fe640 10 client.?.objecter ms_handle_connect 0x7f1db80b2d70 2024-04-29T20:48:24.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.645+0000 7f1d9f7fe640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0018020 con 0x7f1db80b2d70 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: dump: 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.650 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: _finish_auth 0 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db0014680 con 0x7f1db80b2d70 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 5 monclient: authenticate success, global_id 16352 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0014a50 con 0x7f1db80b2d70 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 monclient: _renew_subs 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db8142cf0 con 0x7f1db80b2d70 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1d9f7fe640 10 monclient: dump: 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1db559a640 4 set_mon_vals no callback set 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 client.16352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 monclient: _renew_subs 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db8143c30 con 0x7f1db80b2d70 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 1 librados: init done 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 client.16352.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 15 client.16352.objecter _session_command_op_assign -1 1 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 20 client.16352.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 client.16352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:24.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.649+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1db8142f70 con 0x7f1db80b2d70 2024-04-29T20:48:24.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.670+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1db0005d10 con 0x7f1db80b2d70 2024-04-29T20:48:24.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.670+0000 7f1d9f7fe640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1da406e2d0 0x7f1da4070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.671+0000 7f1db659c640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1da406e2d0 0x7f1da4070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.671+0000 7f1db659c640 10 monclient: get_auth_request con 0x7f1da406e2d0 auth_method 0 2024-04-29T20:48:24.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.671+0000 7f1db659c640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1da406e2d0 0x7f1da4070790 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1dac036000 tx=0x7f1dac03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.671+0000 7f1d9f7fe640 10 client.16352.objecter ms_handle_connect 0x7f1da406e2d0 2024-04-29T20:48:24.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f1db0059050 con 0x7f1db80b2d70 2024-04-29T20:48:24.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 10 client.16352.objecter ms_dispatch 0x7f1db8000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:24.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 3 client.16352.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 3 client.16352.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 10 client.16352.objecter checking command 1 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f1da4075fd0 0x7f1da4078470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 20 client.16352.objecter _get_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 15 client.16352.objecter _session_command_op_remove -1 1 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 20 client.16352.objecter _get_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 20 client.16352.objecter get_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 15 client.16352.objecter _session_command_op_assign 0 1 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 4 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 10 client.16352.objecter _send_command 1 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1db0009a50 con 0x7f1da4075fd0 2024-04-29T20:48:24.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.673+0000 7f1d9f7fe640 20 client.16352.objecter dump_active .. 0 homeless 2024-04-29T20:48:24.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.674+0000 7f1db759e640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f1da4075fd0 0x7f1da4078470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.674+0000 7f1db759e640 10 monclient: get_auth_request con 0x7f1da4075fd0 auth_method 0 2024-04-29T20:48:24.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.675+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7f1db0057250 con 0x7f1db80b2d70 2024-04-29T20:48:24.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.675+0000 7f1d9f7fe640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:24.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.677+0000 7f1d9f7fe640 10 client.16352.objecter ms_handle_connect 0x7f1da4075fd0 2024-04-29T20:48:24.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.677+0000 7f1db759e640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f1da4075fd0 0x7f1da4078470 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.702+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== osd.0 v2:172.21.15.64:6808/2796311601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1db0009a50 con 0x7f1da4075fd0 2024-04-29T20:48:24.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.702+0000 7f1d9f7fe640 10 client.16352.objecter ms_dispatch 0x7f1db8000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:24.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.702+0000 7f1d9f7fe640 10 client.16352.objecter _finish_command 1 = system:0 2024-04-29T20:48:24.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.702+0000 7f1d9f7fe640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 4 2024-04-29T20:48:24.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.702+0000 7f1d9f7fe640 15 client.16352.objecter _session_command_op_remove 0 1 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 10 client.16352.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 15 client.16352.objecter _session_command_op_assign -1 2 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 20 client.16352.objecter _get_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 20 client.16352.objecter _get_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 15 client.16352.objecter _session_command_op_remove -1 2 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 20 client.16352.objecter get_session s=0x7f1da4075900 osd=0 3 2024-04-29T20:48:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 15 client.16352.objecter _session_command_op_assign 0 2 2024-04-29T20:48:24.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 4 2024-04-29T20:48:24.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 10 client.16352.objecter _send_command 2 2024-04-29T20:48:24.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.715+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 --> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1db804fce0 con 0x7f1da4075fd0 2024-04-29T20:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.732+0000 7f1d9f7fe640 1 -- 172.21.15.64:0/951540535 <== osd.0 v2:172.21.15.64:6808/2796311601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1db804fce0 con 0x7f1da4075fd0 2024-04-29T20:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.732+0000 7f1d9f7fe640 10 client.16352.objecter ms_dispatch 0x7f1db8000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.732+0000 7f1d9f7fe640 10 client.16352.objecter _finish_command 2 = system:0 2024-04-29T20:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.732+0000 7f1d9f7fe640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 4 2024-04-29T20:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.732+0000 7f1d9f7fe640 15 client.16352.objecter _session_command_op_remove 0 2 2024-04-29T20:48:24.733 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:24.733 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:24.733 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 10 librados: watch_flush enter 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 10 librados: watch_flush exit 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 10 client.16352.objecter close_session for osd.0 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f1da4075fd0 msgr2=0x7f1da4078470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:6808/2796311601,v1:172.21.15.64:6810/2796311601] conn(0x7f1da4075fd0 0x7f1da4078470 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 20 client.16352.objecter put_session s=0x7f1da4075900 osd=0 1 2024-04-29T20:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 20 client.16352.objecter shutdown clearing up homeless session... 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 10 client.16352.objecter successfully canceled tick 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1da406e2d0 msgr2=0x7f1da4070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1da406e2d0 0x7f1da4070790 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1dac036000 tx=0x7f1dac03c040 comp rx=0 tx=0).stop 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 10 monclient: shutdown 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b2d70 msgr2=0x7f1db813ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 --2- 172.21.15.64:0/951540535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1db80b2d70 0x7f1db813ed80 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009f60 tx=0x7f1db0009f90 comp rx=0 tx=0).stop 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1db759e640 1 -- 172.21.15.64:0/951540535 reap_dead start 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 shutdown_connections 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 >> 172.21.15.64:0/951540535 conn(0x7f1db806c9d0 msgr2=0x7f1db813c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 shutdown_connections 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 -- 172.21.15.64:0/951540535 wait complete. 2024-04-29T20:48:24.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.733+0000 7f1dbd08c640 1 librados: shutdown 2024-04-29T20:48:24.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.850+0000 7febea9f4640 10 monclient: get_monmap_and_config 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.850+0000 7febea9f4640 10 monclient: build_initial_monmap 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.850+0000 7febea9f4640 10 monclient: monmap: 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:24.851178+0000 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:24.851178+0000 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:24.850 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:24.851 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:24.851 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:24.851 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:24.851 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 1 Processor -- start 2024-04-29T20:48:24.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 1 -- start start 2024-04-29T20:48:24.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient: init 2024-04-29T20:48:24.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:24.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe4146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient(hunting): picked mon.noname-b con 0x7febe40b22a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febe40b2c20 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febe40aa6a0 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe40ac1f0 con 0x7febe40b2c20 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe40ac4d0 con 0x7febe40aa6a0 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7febe40b33d0 con 0x7febe40b22a0 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.851+0000 7febea9f4640 10 monclient(hunting): authenticate will time out at 3054.011315s 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe3fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe4146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe37fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7febe40b2c20 0x7febe40b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60472/0 (socket says 172.21.15.64:60472) 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe37fe640 1 -- 172.21.15.64:0/1092341957 learned_addr learned my addr 172.21.15.64:0/1092341957 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe3fff640 10 monclient(hunting): get_auth_request con 0x7febe40b22a0 auth_method 0 2024-04-29T20:48:24.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe3fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.852+0000 7febe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.853+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.853+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.853+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.854+0000 7febe2ffd640 1 -- 172.21.15.64:0/1092341957 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3106946579 0 0) 0x7febe40ac4d0 con 0x7febe40aa6a0 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.854+0000 7febe2ffd640 10 monclient(hunting): _negotiate 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.854+0000 7febe2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.854+0000 7febe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.854+0000 7febe2ffd640 10 monclient(hunting): my global_id is 22141 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe2ffd640 1 -- 172.21.15.64:0/1092341957 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febcc003520 con 0x7febe40aa6a0 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 10 monclient(hunting): handle_auth_done global_id 16355 payload 274 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 1 -- 172.21.15.64:0/1092341957 >> v1:172.21.15.167:6789/0 conn(0x7febe40aa6a0 legacy=0x7febe414ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 1 -- 172.21.15.64:0/1092341957 >> v1:172.21.15.64:6789/0 conn(0x7febe40b2c20 legacy=0x7febe40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 10 monclient: _finish_hunting 0 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 1 monclient: found mon.noname-b 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe2ffd640 1 -- 172.21.15.64:0/1092341957 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 93522776 0 0) 0x7febe40ac1f0 con 0x7febe40b2c20 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe3fff640 1 -- 172.21.15.64:0/1092341957 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe40b3040 con 0x7febe40b22a0 2024-04-29T20:48:24.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.855+0000 7febe2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-29T20:48:24.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.857+0000 7febe3fff640 1 --2- 172.21.15.64:0/1092341957 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe4146a40 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7febe40b3550 tx=0x7febd402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49dcf275e9f1ca79 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 1 -- 172.21.15.64:0/1092341957 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4035020 con 0x7febe40b22a0 2024-04-29T20:48:24.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:24.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 10 monclient: dump: 2024-04-29T20:48:24.859 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.859 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 10 monclient: _finish_auth 0 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febea9f4640 5 monclient: authenticate success, global_id 16355 2024-04-29T20:48:24.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.859+0000 7febea9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:24.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febe2ffd640 1 -- 172.21.15.64:0/1092341957 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd4031890 con 0x7febe40b22a0 2024-04-29T20:48:24.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febe2ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:24.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febe2ffd640 1 -- 172.21.15.64:0/1092341957 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4031c40 con 0x7febe40b22a0 2024-04-29T20:48:24.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febea9f4640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:24.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febea9f4640 4 set_mon_vals no callback set 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.861+0000 7febe2ffd640 10 monclient: dump: 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.862 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 10 monclient: shutdown 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 -- 172.21.15.64:0/1092341957 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 msgr2=0x7febe4146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 --2- 172.21.15.64:0/1092341957 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe4146a40 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7febe40b3550 tx=0x7febd402b930 comp rx=0 tx=0).stop 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 -- 172.21.15.64:0/1092341957 shutdown_connections 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 --2- 172.21.15.64:0/1092341957 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe4146a40 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 -- 172.21.15.64:0/1092341957 >> 172.21.15.64:0/1092341957 conn(0x7febe406c940 msgr2=0x7febe406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 -- 172.21.15.64:0/1092341957 shutdown_connections 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.862+0000 7febea9f4640 1 -- 172.21.15.64:0/1092341957 wait complete. 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.863+0000 7febea9f4640 10 monclient: build_initial_monmap 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.863+0000 7febea9f4640 10 monclient: monmap: 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:24.864 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 librados: starting msgr at 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 librados: starting objecter 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 Processor -- start 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 -- start start 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 librados: setting wanted keys 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 librados: calling monclient init 2024-04-29T20:48:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient: init 2024-04-29T20:48:24.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:24.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:48:24.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe40aa6a0 0x7febe41dba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): picked mon.noname-a con 0x7febe40aa6a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe41368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): picked mon.noname-c con 0x7febe40b22a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febe40b2c20 0x7febe4136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): picked mon.noname-b con 0x7febe40b2c20 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7febe40aacb0 con 0x7febe40aa6a0 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7febe40aab30 con 0x7febe40b2c20 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7febe414f4f0 con 0x7febe40b22a0 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.864+0000 7febea9f4640 10 monclient(hunting): authenticate will time out at 3054.024440s 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe41368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe40aa6a0 0x7febe41dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe41368e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45608/0 (socket says 172.21.15.64:45608) 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 1 -- 172.21.15.64:0/2216328329 learned_addr learned my addr 172.21.15.64:0/2216328329 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 10 monclient(hunting): get_auth_request con 0x7febe40b22a0 auth_method 0 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.865+0000 7febe8f91640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febe40b2c20 0x7febe4136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): get_auth_request con 0x7febe40aa6a0 auth_method 0 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): get_auth_request con 0x7febe40b2c20 auth_method 0 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:24.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:24.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.866+0000 7febe8f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 10 monclient(hunting): handle_auth_done global_id 22144 payload 824 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 1 -- 172.21.15.64:0/2216328329 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 msgr2=0x7febe41368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe41368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 1 -- 172.21.15.64:0/2216328329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe40aa6a0 msgr2=0x7febe41dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe40aa6a0 0x7febe41dba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 10 monclient: _finish_hunting 0 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 1 monclient: found mon.noname-b 2024-04-29T20:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:24.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:24.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.869+0000 7febe8f91640 1 -- 172.21.15.64:0/2216328329 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe41dc050 con 0x7febe40b2c20 2024-04-29T20:48:24.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.871+0000 7febe8f91640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febe40b2c20 0x7febe4136e20 secure :-1 s=READY pgs=5622 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c6f0 tx=0x7febdc00cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.871+0000 7febe0ff9640 10 client.?.objecter ms_handle_connect 0x7febe40b2c20 2024-04-29T20:48:24.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.871+0000 7febe0ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc003970 con 0x7febe40b2c20 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: dump: 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: _finish_auth 0 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febdc003b30 con 0x7febe40b2c20 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 5 monclient: authenticate success, global_id 22144 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 10 monclient: _renew_subs 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe413ced0 con 0x7febe40b2c20 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc00a850 con 0x7febe40b2c20 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe0ff9640 10 monclient: dump: 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febe27fc640 4 set_mon_vals no callback set 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 10 client.22144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 10 monclient: _renew_subs 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe4134c30 con 0x7febe40b2c20 2024-04-29T20:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.875+0000 7febea9f4640 1 librados: init done 2024-04-29T20:48:24.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.876+0000 7febea9f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:24.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.876+0000 7febea9f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:24.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.876+0000 7febea9f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:24.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.876+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe404fce0 con 0x7febe40b2c20 2024-04-29T20:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.893+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febdc00a9f0 con 0x7febe40b2c20 2024-04-29T20:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.894+0000 7febe0ff9640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd006e280 0x7febd0070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.894+0000 7febe3fff640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd006e280 0x7febd0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.894+0000 7febe3fff640 10 monclient: get_auth_request con 0x7febd006e280 auth_method 0 2024-04-29T20:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.894+0000 7febe3fff640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd006e280 0x7febd0070740 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7febd4036000 tx=0x7febd4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.894+0000 7febe0ff9640 10 client.22144.objecter ms_handle_connect 0x7febd006e280 2024-04-29T20:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.896+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7febdc058280 con 0x7febe40b2c20 2024-04-29T20:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.896+0000 7febe0ff9640 10 client.22144.objecter ms_dispatch 0x7febe4000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.896+0000 7febe0ff9640 3 client.22144.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.896+0000 7febe0ff9640 3 client.22144.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:24.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:24.897+0000 7febe0ff9640 20 client.22144.objecter dump_active .. 0 homeless 2024-04-29T20:48:25.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.140+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febdc05a1e0 con 0x7febe40b2c20 2024-04-29T20:48:25.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.140+0000 7febe0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:25.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.140+0000 7febe0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:25.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.332+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7febe41dc220 con 0x7febd006e280 2024-04-29T20:48:25.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.333+0000 7febe0ff9640 1 -- 172.21.15.64:0/2216328329 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7febe41dc220 con 0x7febd006e280 2024-04-29T20:48:25.333 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 0 to deep-scrub 2024-04-29T20:48:25.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 10 librados: watch_flush enter 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 10 librados: watch_flush exit 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 20 client.22144.objecter shutdown clearing up homeless session... 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 10 client.22144.objecter successfully canceled tick 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd006e280 msgr2=0x7febd0070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd006e280 0x7febd0070740 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7febd4036000 tx=0x7febd4008030 comp rx=0 tx=0).stop 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 10 monclient: shutdown 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febe40b2c20 msgr2=0x7febe4136e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febe40b2c20 0x7febe4136e20 secure :-1 s=READY pgs=5622 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c6f0 tx=0x7febdc00cbb0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 shutdown_connections 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7febd006e280 0x7febd0070740 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7febe40b22a0 0x7febe41368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febe40b2c20 0x7febe4136e20 unknown :-1 s=CLOSED pgs=5622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 --2- 172.21.15.64:0/2216328329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe40aa6a0 0x7febe41dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 >> 172.21.15.64:0/2216328329 conn(0x7febe406c940 msgr2=0x7febe41453f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:25.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 shutdown_connections 2024-04-29T20:48:25.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 -- 172.21.15.64:0/2216328329 wait complete. 2024-04-29T20:48:25.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.335+0000 7febea9f4640 1 librados: shutdown 2024-04-29T20:48:25.347 INFO:tasks.ceph:Scrubbing osd.1 2024-04-29T20:48:25.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.454+0000 7fc7d7c9a640 10 monclient: get_monmap_and_config 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.454+0000 7fc7d7c9a640 10 monclient: build_initial_monmap 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.454+0000 7fc7d7c9a640 10 monclient: monmap: 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:25.455377+0000 2024-04-29T20:48:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:25.455377+0000 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:25.455 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.455+0000 7fc7d7c9a640 1 Processor -- start 2024-04-29T20:48:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.455+0000 7fc7d7c9a640 1 -- start start 2024-04-29T20:48:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.455+0000 7fc7d7c9a640 10 monclient: init 2024-04-29T20:48:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-29T20:48:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d006cdb0 0x7fc7d01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7d006cdb0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7d00b3270 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc7d0140d00 0x7fc7d0147160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7d0140d00 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d0141170 con 0x7fc7d00b3270 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d006d190 con 0x7fc7d0140d00 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d01476a0 con 0x7fc7d006cdb0 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.456+0000 7fc7d7c9a640 10 monclient(hunting): authenticate will time out at 3054.616044s 2024-04-29T20:48:25.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.484+0000 7fc7d6237640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc7d0140d00 0x7fc7d0147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d5235640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc7d00b3270 0x7fc7d0146a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:60166/0 (socket says 172.21.15.64:60166) 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d5235640 1 -- 172.21.15.64:0/1841014472 learned_addr learned my addr 172.21.15.64:0/1841014472 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): get_auth_request con 0x7fc7d0140d00 auth_method 0 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.485+0000 7fc7d6237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 10 monclient(hunting): handle_auth_done global_id 22147 payload 274 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 1 -- 172.21.15.64:0/1841014472 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d006cdb0 msgr2=0x7fc7d01407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 1 --2- 172.21.15.64:0/1841014472 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d006cdb0 0x7fc7d01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 1 -- 172.21.15.64:0/1841014472 >> v1:172.21.15.167:6789/0 conn(0x7fc7d00b3270 legacy=0x7fc7d0146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 10 monclient: _finish_hunting 0 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 1 monclient: found mon.noname-c 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.488+0000 7fc7d6237640 1 -- 172.21.15.64:0/1841014472 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d00abff0 con 0x7fc7d0140d00 2024-04-29T20:48:25.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.494+0000 7fc7d6237640 1 --2- 172.21.15.64:0/1841014472 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d0140d00 0x7fc7d0147160 secure :-1 s=READY pgs=5624 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5d8baa1f829a08c server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 1 -- 172.21.15.64:0/1841014472 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8039070 con 0x7fc7d0140d00 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: dump: 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: _finish_auth 0 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: _check_auth_tickets 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 1 -- 172.21.15.64:0/1841014472 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c802cbc0 con 0x7fc7d0140d00 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:25.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 1 -- 172.21.15.64:0/1841014472 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8034440 con 0x7fc7d0140d00 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d4a34640 10 monclient: dump: 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d7c9a640 5 monclient: authenticate success, global_id 22147 2024-04-29T20:48:25.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d7c9a640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:25.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.498+0000 7fc7d7c9a640 4 set_mon_vals no callback set 2024-04-29T20:48:25.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 10 monclient: shutdown 2024-04-29T20:48:25.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:25.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1841014472 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d0140d00 msgr2=0x7fc7d0147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 --2- 172.21.15.64:0/1841014472 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d0140d00 0x7fc7d0147160 secure :-1 s=READY pgs=5624 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c802be10 comp rx=0 tx=0).stop 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1841014472 shutdown_connections 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 --2- 172.21.15.64:0/1841014472 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d0140d00 0x7fc7d0147160 unknown :-1 s=CLOSED pgs=5624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 --2- 172.21.15.64:0/1841014472 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d006cdb0 0x7fc7d01407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1841014472 >> 172.21.15.64:0/1841014472 conn(0x7fc7d00b9030 msgr2=0x7fc7d00b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1841014472 shutdown_connections 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1841014472 wait complete. 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 10 monclient: build_initial_monmap 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.499+0000 7fc7d7c9a640 10 monclient: monmap: 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:25.504 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 librados: starting msgr at 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 librados: starting objecter 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 Processor -- start 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 -- start start 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 librados: setting wanted keys 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 librados: calling monclient init 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient: init 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d006cdb0 0x7fc7d0136570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): picked mon.noname-b con 0x7fc7d006cdb0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d00b3270 0x7fc7d01d8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): picked mon.noname-c con 0x7fc7d00b3270 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7d0140d00 0x7fc7d0134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): picked mon.noname-a con 0x7fc7d0140d00 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d00aace0 con 0x7fc7d0140d00 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d00ab5d0 con 0x7fc7d006cdb0 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d00aae60 con 0x7fc7d00b3270 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.500+0000 7fc7d7c9a640 10 monclient(hunting): authenticate will time out at 3054.660428s 2024-04-29T20:48:25.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5235640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d00b3270 0x7fc7d01d8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d006cdb0 0x7fc7d0136570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d006cdb0 0x7fc7d0136570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56562/0 (socket says 172.21.15.64:56562) 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5235640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d00b3270 0x7fc7d01d8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45614/0 (socket says 172.21.15.64:45614) 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 1 -- 172.21.15.64:0/1509261103 learned_addr learned my addr 172.21.15.64:0/1509261103 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 10 monclient(hunting): get_auth_request con 0x7fc7d006cdb0 auth_method 0 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5a36640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5235640 10 monclient(hunting): get_auth_request con 0x7fc7d00b3270 auth_method 0 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5235640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5235640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.501+0000 7fc7d5235640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d6237640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7d0140d00 0x7fc7d0134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d5a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d5a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d5a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d6237640 10 monclient(hunting): get_auth_request con 0x7fc7d0140d00 auth_method 0 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d6237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d6237640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d6237640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d5235640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d5235640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.502+0000 7fc7d5235640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.503+0000 7fc7d6237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.503+0000 7fc7d6237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.503+0000 7fc7d6237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 10 monclient(hunting): handle_auth_done global_id 22150 payload 824 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 1 -- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d00b3270 msgr2=0x7fc7d01d8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc7d00b3270 0x7fc7d01d8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 1 -- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7d0140d00 msgr2=0x7fc7d0134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7d0140d00 0x7fc7d0134340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 10 monclient: _finish_hunting 0 2024-04-29T20:48:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 1 monclient: found mon.noname-b 2024-04-29T20:48:25.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.504+0000 7fc7d5a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:25.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.505+0000 7fc7d5a36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.505+0000 7fc7d5a36640 1 -- 172.21.15.64:0/1509261103 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d013c960 con 0x7fc7d006cdb0 2024-04-29T20:48:25.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.507+0000 7fc7d5a36640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d006cdb0 0x7fc7d0136570 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00b4a0 tx=0x7fc7cc00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.507+0000 7fc7c67fc640 10 client.?.objecter ms_handle_connect 0x7fc7d006cdb0 2024-04-29T20:48:25.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.507+0000 7fc7c67fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:25.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc014070 con 0x7fc7d006cdb0 2024-04-29T20:48:25.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:25.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: dump: 2024-04-29T20:48:25.511 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.511 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: _finish_auth 0 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: _check_auth_tickets 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7cc010610 con 0x7fc7d006cdb0 2024-04-29T20:48:25.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 5 monclient: authenticate success, global_id 22150 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 10 monclient: _renew_subs 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc010a00 con 0x7fc7d006cdb0 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d013d850 con 0x7fc7d006cdb0 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c67fc640 10 monclient: dump: 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.513 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7c7fff640 4 set_mon_vals no callback set 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 10 client.22150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 10 monclient: _renew_subs 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d01d6140 con 0x7fc7d006cdb0 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.510+0000 7fc7d7c9a640 1 librados: init done 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.511+0000 7fc7d7c9a640 10 client.22150.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.511+0000 7fc7d7c9a640 15 client.22150.objecter _session_command_op_assign -1 1 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.511+0000 7fc7d7c9a640 20 client.22150.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.511+0000 7fc7d7c9a640 10 client.22150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:25.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.511+0000 7fc7d7c9a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.511+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7d004fce0 con 0x7fc7d006cdb0 2024-04-29T20:48:25.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.530+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc7cc017070 con 0x7fc7d006cdb0 2024-04-29T20:48:25.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.531+0000 7fc7c67fc640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc7b406e2d0 0x7fc7b4070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.531+0000 7fc7d5235640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc7b406e2d0 0x7fc7b4070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.531+0000 7fc7d5235640 10 monclient: get_auth_request con 0x7fc7b406e2d0 auth_method 0 2024-04-29T20:48:25.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.532+0000 7fc7d5235640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc7b406e2d0 0x7fc7b4070790 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00b4710 tx=0x7fc7c0007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.532+0000 7fc7c67fc640 10 client.22150.objecter ms_handle_connect 0x7fc7b406e2d0 2024-04-29T20:48:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7fc7cc0578a0 con 0x7fc7d006cdb0 2024-04-29T20:48:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 10 client.22150.objecter ms_dispatch 0x7fc7d0000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 3 client.22150.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 3 client.22150.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 10 client.22150.objecter checking command 1 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7fc7b4075fd0 0x7fc7b4078470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 20 client.22150.objecter _get_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 15 client.22150.objecter _session_command_op_remove -1 1 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 20 client.22150.objecter _get_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 20 client.22150.objecter get_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 15 client.22150.objecter _session_command_op_assign 1 1 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 4 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 10 client.22150.objecter _send_command 1 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 --> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7b4078b40 con 0x7fc7b4075fd0 2024-04-29T20:48:25.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.533+0000 7fc7c67fc640 20 client.22150.objecter dump_active .. 0 homeless 2024-04-29T20:48:25.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.534+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7fc7cc055aa0 con 0x7fc7d006cdb0 2024-04-29T20:48:25.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.534+0000 7fc7c67fc640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:25.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.534+0000 7fc7d6237640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7fc7b4075fd0 0x7fc7b4078470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.535+0000 7fc7d6237640 10 monclient: get_auth_request con 0x7fc7b4075fd0 auth_method 0 2024-04-29T20:48:25.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.538+0000 7fc7d6237640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7fc7b4075fd0 0x7fc7b4078470 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.538+0000 7fc7c67fc640 10 client.22150.objecter ms_handle_connect 0x7fc7b4075fd0 2024-04-29T20:48:25.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.563+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== osd.1 v2:172.21.15.64:6809/3421706551 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7b4078b40 con 0x7fc7b4075fd0 2024-04-29T20:48:25.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.563+0000 7fc7c67fc640 10 client.22150.objecter ms_dispatch 0x7fc7d0000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:25.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.563+0000 7fc7c67fc640 10 client.22150.objecter _finish_command 1 = system:0 2024-04-29T20:48:25.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.563+0000 7fc7c67fc640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 4 2024-04-29T20:48:25.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.563+0000 7fc7c67fc640 15 client.22150.objecter _session_command_op_remove 1 1 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 10 client.22150.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 15 client.22150.objecter _session_command_op_assign -1 2 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 20 client.22150.objecter _get_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 20 client.22150.objecter _get_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 15 client.22150.objecter _session_command_op_remove -1 2 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 20 client.22150.objecter get_session s=0x7fc7b4075900 osd=1 3 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 15 client.22150.objecter _session_command_op_assign 1 2 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 4 2024-04-29T20:48:25.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 10 client.22150.objecter _send_command 2 2024-04-29T20:48:25.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.576+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 --> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc7d0134ad0 con 0x7fc7b4075fd0 2024-04-29T20:48:25.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7c67fc640 1 -- 172.21.15.64:0/1509261103 <== osd.1 v2:172.21.15.64:6809/3421706551 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc7d0134ad0 con 0x7fc7b4075fd0 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7c67fc640 10 client.22150.objecter ms_dispatch 0x7fc7d0000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7c67fc640 10 client.22150.objecter _finish_command 2 = system:0 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7c67fc640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 4 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7c67fc640 15 client.22150.objecter _session_command_op_remove 1 2 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:25.590 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:25.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 10 librados: watch_flush enter 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 10 librados: watch_flush exit 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 10 client.22150.objecter close_session for osd.1 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7fc7b4075fd0 msgr2=0x7fc7b4078470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.64:6809/3421706551,v1:172.21.15.64:6811/3421706551] conn(0x7fc7b4075fd0 0x7fc7b4078470 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 20 client.22150.objecter put_session s=0x7fc7b4075900 osd=1 1 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 20 client.22150.objecter shutdown clearing up homeless session... 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 10 client.22150.objecter successfully canceled tick 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc7b406e2d0 msgr2=0x7fc7b4070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc7b406e2d0 0x7fc7b4070790 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00b4710 tx=0x7fc7c0007e80 comp rx=0 tx=0).stop 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 10 monclient: shutdown 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d006cdb0 msgr2=0x7fc7d0136570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 --2- 172.21.15.64:0/1509261103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7d006cdb0 0x7fc7d0136570 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00b4a0 tx=0x7fc7cc00b960 comp rx=0 tx=0).stop 2024-04-29T20:48:25.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d6237640 1 -- 172.21.15.64:0/1509261103 reap_dead start 2024-04-29T20:48:25.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 shutdown_connections 2024-04-29T20:48:25.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 >> 172.21.15.64:0/1509261103 conn(0x7fc7d00b9030 msgr2=0x7fc7d0146f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:25.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 shutdown_connections 2024-04-29T20:48:25.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 -- 172.21.15.64:0/1509261103 wait complete. 2024-04-29T20:48:25.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.590+0000 7fc7d7c9a640 1 librados: shutdown 2024-04-29T20:48:25.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-29T20:48:25.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.703+0000 7f2a958d1640 10 monclient: get_monmap_and_config 2024-04-29T20:48:25.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.703+0000 7f2a958d1640 10 monclient: build_initial_monmap 2024-04-29T20:48:25.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.703+0000 7f2a958d1640 10 monclient: monmap: 2024-04-29T20:48:25.703 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:25.703 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:25.704334+0000 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:25.704334+0000 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:25.704 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 Processor -- start 2024-04-29T20:48:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 -- start start 2024-04-29T20:48:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient: init 2024-04-29T20:48:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-29T20:48:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a900aa460 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a900aa840 0x7f2a900b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): picked mon.noname-b con 0x7f2a900aa840 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2a90143430 0x7f2a900b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): picked mon.noname-c con 0x7f2a90143430 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a900ab3c0 con 0x7f2a900aa460 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f2a900b34b0 con 0x7f2a90143430 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f2a901438c0 con 0x7f2a900aa840 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.704+0000 7f2a958d1640 10 monclient(hunting): authenticate will time out at 3054.864295s 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8e7fc640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a900aa840 0x7f2a900b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8f7fe640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f2a90143430 0x7f2a900b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8effd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2a900aa460 0x7f2a900b2da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60488/0 (socket says 172.21.15.64:60488) 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8effd640 1 -- 172.21.15.64:0/1700869068 learned_addr learned my addr 172.21.15.64:0/1700869068 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a90143430 auth_method 0 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8e7fc640 10 monclient(hunting): get_auth_request con 0x7f2a900aa840 auth_method 0 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.705+0000 7f2a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.706+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.706+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.706+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.706+0000 7f2a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.706+0000 7f2a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.706+0000 7f2a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_done global_id 22153 payload 274 2024-04-29T20:48:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 1 -- 172.21.15.64:0/1700869068 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a900aa840 msgr2=0x7f2a900b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 1 --2- 172.21.15.64:0/1700869068 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a900aa840 0x7f2a900b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 1 -- 172.21.15.64:0/1700869068 >> v1:172.21.15.64:6789/0 conn(0x7f2a900aa460 legacy=0x7f2a900b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:48:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 1 monclient: found mon.noname-c 2024-04-29T20:48:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.707+0000 7f2a8f7fe640 1 -- 172.21.15.64:0/1700869068 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a900b9730 con 0x7f2a90143430 2024-04-29T20:48:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.709+0000 7f2a8f7fe640 1 --2- 172.21.15.64:0/1700869068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a90143430 0x7f2a900b1e90 secure :-1 s=READY pgs=5626 cs=0 l=1 rev1=1 crypto rx=0x7f2a880097d0 tx=0x7f2a8802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fa0ecdf3dbb9bac server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 1 -- 172.21.15.64:0/1700869068 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88039070 con 0x7f2a90143430 2024-04-29T20:48:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: dump: 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: _finish_auth 0 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: _check_auth_tickets 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 1 -- 172.21.15.64:0/1700869068 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a880316f0 con 0x7f2a90143430 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 1 -- 172.21.15.64:0/1700869068 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88031ac0 con 0x7f2a90143430 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a8dffb640 10 monclient: dump: 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.716 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a958d1640 5 monclient: authenticate success, global_id 22153 2024-04-29T20:48:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a958d1640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.713+0000 7f2a958d1640 4 set_mon_vals no callback set 2024-04-29T20:48:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 10 monclient: shutdown 2024-04-29T20:48:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 -- 172.21.15.64:0/1700869068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a90143430 msgr2=0x7f2a900b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1700869068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a90143430 0x7f2a900b1e90 secure :-1 s=READY pgs=5626 cs=0 l=1 rev1=1 crypto rx=0x7f2a880097d0 tx=0x7f2a8802b790 comp rx=0 tx=0).stop 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 -- 172.21.15.64:0/1700869068 shutdown_connections 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1700869068 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a90143430 0x7f2a900b1e90 unknown :-1 s=CLOSED pgs=5626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1700869068 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a900aa840 0x7f2a900b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 -- 172.21.15.64:0/1700869068 >> 172.21.15.64:0/1700869068 conn(0x7f2a9006c960 msgr2=0x7f2a9006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 -- 172.21.15.64:0/1700869068 shutdown_connections 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.714+0000 7f2a958d1640 1 -- 172.21.15.64:0/1700869068 wait complete. 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 10 monclient: build_initial_monmap 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 10 monclient: monmap: 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 1 librados: starting msgr at 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 1 librados: starting objecter 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 1 Processor -- start 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 1 -- start start 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 1 librados: setting wanted keys 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 1 librados: calling monclient init 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.715+0000 7f2a958d1640 10 monclient: init 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a900aa460 0x7f2a9013f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): picked mon.noname-a con 0x7f2a900aa460 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a90143430 0x7f2a900aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): picked mon.noname-c con 0x7f2a90143430 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a900b0f40 0x7f2a900ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): picked mon.noname-b con 0x7f2a900b0f40 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2a900b2550 con 0x7f2a900aa460 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2a900b23d0 con 0x7f2a900b0f40 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f2a900b26d0 con 0x7f2a90143430 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a958d1640 10 monclient(hunting): authenticate will time out at 3054.875975s 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a8e7fc640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a90143430 0x7f2a900aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.716+0000 7f2a8effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a900aa460 0x7f2a9013f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a900b0f40 0x7f2a900ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a90143430 0x7f2a900aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45634/0 (socket says 172.21.15.64:45634) 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 1 -- 172.21.15.64:0/1297238923 learned_addr learned my addr 172.21.15.64:0/1297238923 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): get_auth_request con 0x7f2a90143430 auth_method 0 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a900b0f40 auth_method 0 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8effd640 10 monclient(hunting): get_auth_request con 0x7f2a900aa460 auth_method 0 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8effd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:25.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.717+0000 7f2a8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:25.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.718+0000 7f2a8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 10 monclient(hunting): handle_auth_done global_id 22156 payload 824 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 1 -- 172.21.15.64:0/1297238923 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a90143430 msgr2=0x7f2a900aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a90143430 0x7f2a900aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 1 -- 172.21.15.64:0/1297238923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a900aa460 msgr2=0x7f2a9013f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a900aa460 0x7f2a9013f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 1 monclient: found mon.noname-b 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.720+0000 7f2a8f7fe640 1 -- 172.21.15.64:0/1297238923 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a900b0b20 con 0x7f2a900b0f40 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.722+0000 7f2a8f7fe640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a900b0f40 0x7f2a900ae7a0 secure :-1 s=READY pgs=5627 cs=0 l=1 rev1=1 crypto rx=0x7f2a900b9c30 tx=0x7f2a880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.722+0000 7f2a77fff640 10 client.?.objecter ms_handle_connect 0x7f2a900b0f40 2024-04-29T20:48:25.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.722+0000 7f2a77fff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:25.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.725+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88039040 con 0x7f2a900b0f40 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.725+0000 7f2a77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.725+0000 7f2a77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.725+0000 7f2a77fff640 10 monclient: dump: 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 10 monclient: _finish_auth 0 2024-04-29T20:48:25.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 10 monclient: _check_auth_tickets 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a88031ea0 con 0x7f2a900b0f40 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a880346c0 con 0x7f2a900b0f40 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 5 monclient: authenticate success, global_id 22156 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 10 monclient: _renew_subs 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a900b17f0 con 0x7f2a900b0f40 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a8d7fa640 4 set_mon_vals no callback set 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a77fff640 10 monclient: dump: 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:25.728 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 10 client.22156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 10 monclient: _renew_subs 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a900b13d0 con 0x7f2a900b0f40 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.726+0000 7f2a958d1640 1 librados: init done 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.727+0000 7f2a958d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.727+0000 7f2a958d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:25.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.727+0000 7f2a958d1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:25.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.727+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a900b1600 con 0x7f2a900b0f40 2024-04-29T20:48:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.977+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a88046020 con 0x7f2a900b0f40 2024-04-29T20:48:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.978+0000 7f2a77fff640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2a7c06e2d0 0x7f2a7c070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.978+0000 7f2a8effd640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2a7c06e2d0 0x7f2a7c070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.978+0000 7f2a8effd640 10 monclient: get_auth_request con 0x7f2a7c06e2d0 auth_method 0 2024-04-29T20:48:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.979+0000 7f2a77fff640 10 client.22156.objecter ms_handle_connect 0x7f2a7c06e2d0 2024-04-29T20:48:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.979+0000 7f2a8effd640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2a7c06e2d0 0x7f2a7c070790 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2a800012e0 tx=0x7f2a80001990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.982+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f2a88079320 con 0x7f2a900b0f40 2024-04-29T20:48:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.982+0000 7f2a77fff640 10 client.22156.objecter ms_dispatch 0x7f2a90000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.982+0000 7f2a77fff640 3 client.22156.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.982+0000 7f2a77fff640 3 client.22156.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.982+0000 7f2a77fff640 20 client.22156.objecter dump_active .. 0 homeless 2024-04-29T20:48:25.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.992+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a8807b280 con 0x7f2a900b0f40 2024-04-29T20:48:25.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.992+0000 7f2a77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:25.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:25.992+0000 7f2a77fff640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.184+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f2a900b0970 con 0x7f2a7c06e2d0 2024-04-29T20:48:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.185+0000 7f2a77fff640 1 -- 172.21.15.64:0/1297238923 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2a900b0970 con 0x7f2a7c06e2d0 2024-04-29T20:48:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 1 to deep-scrub 2024-04-29T20:48:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 10 librados: watch_flush enter 2024-04-29T20:48:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 10 librados: watch_flush exit 2024-04-29T20:48:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 20 client.22156.objecter shutdown clearing up homeless session... 2024-04-29T20:48:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 10 client.22156.objecter successfully canceled tick 2024-04-29T20:48:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2a7c06e2d0 msgr2=0x7f2a7c070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2a7c06e2d0 0x7f2a7c070790 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2a800012e0 tx=0x7f2a80001990 comp rx=0 tx=0).stop 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 10 monclient: shutdown 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a900b0f40 msgr2=0x7f2a900ae7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a900b0f40 0x7f2a900ae7a0 secure :-1 s=READY pgs=5627 cs=0 l=1 rev1=1 crypto rx=0x7f2a900b9c30 tx=0x7f2a880046c0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 shutdown_connections 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f2a7c06e2d0 0x7f2a7c070790 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f2a90143430 0x7f2a900aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a900b0f40 0x7f2a900ae7a0 unknown :-1 s=CLOSED pgs=5627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 --2- 172.21.15.64:0/1297238923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a900aa460 0x7f2a9013f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 >> 172.21.15.64:0/1297238923 conn(0x7f2a9006c960 msgr2=0x7f2a900b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 shutdown_connections 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 -- 172.21.15.64:0/1297238923 wait complete. 2024-04-29T20:48:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.187+0000 7f2a958d1640 1 librados: shutdown 2024-04-29T20:48:26.199 INFO:tasks.ceph:Scrubbing osd.2 2024-04-29T20:48:26.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.305+0000 7fc509b3d640 10 monclient: get_monmap_and_config 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.305+0000 7fc509b3d640 10 monclient: build_initial_monmap 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.305+0000 7fc509b3d640 10 monclient: monmap: 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:26.306170+0000 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:26.306170+0000 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:26.306 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 1 Processor -- start 2024-04-29T20:48:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 1 -- start start 2024-04-29T20:48:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient: init 2024-04-29T20:48:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-04-29T20:48:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fc50406d0e0 0x7fc5040aae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fc50406d0e0 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc504146a50 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc504146ec0 0x7fc50414dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fc504146ec0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5040ac310 con 0x7fc504146a50 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.306+0000 7fc509b3d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fc50406d4c0 con 0x7fc50406d0e0 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc509b3d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc509b3d640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc5040ac5f0 con 0x7fc504146ec0 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc509b3d640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc509b3d640 10 monclient(hunting): authenticate will time out at 3055.466576s 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc503fff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc504146ec0 0x7fc50414dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc4fbfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fc504146a50 0x7fc504140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60500/0 (socket says 172.21.15.64:60500) 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc4fbfff640 1 -- 172.21.15.64:0/1726290374 learned_addr learned my addr 172.21.15.64:0/1726290374 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.307+0000 7fc5037fe640 1 --2- 172.21.15.64:0/1726290374 >> v2:172.21.15.167:3300/0 conn(0x7fc50406d0e0 0x7fc5040aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): get_auth_request con 0x7fc50406d0e0 auth_method 0 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): get_auth_request con 0x7fc504146ec0 auth_method 0 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc5037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.308+0000 7fc503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 10 monclient(hunting): handle_auth_done global_id 22159 payload 274 2024-04-29T20:48:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 1 -- 172.21.15.64:0/1726290374 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc504146ec0 msgr2=0x7fc50414dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 1 --2- 172.21.15.64:0/1726290374 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc504146ec0 0x7fc50414dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 1 -- 172.21.15.64:0/1726290374 >> v1:172.21.15.64:6789/0 conn(0x7fc504146a50 legacy=0x7fc504140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 10 monclient: _finish_hunting 0 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 1 monclient: found mon.noname-c 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.310+0000 7fc5037fe640 1 -- 172.21.15.64:0/1726290374 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5040aa940 con 0x7fc50406d0e0 2024-04-29T20:48:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.312+0000 7fc5037fe640 1 --2- 172.21.15.64:0/1726290374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc50406d0e0 0x7fc5040aae60 secure :-1 s=READY pgs=5628 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40097a0 tx=0x7fc4f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb7222227679491e server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 1 -- 172.21.15.64:0/1726290374 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4035020 con 0x7fc50406d0e0 2024-04-29T20:48:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: dump: 2024-04-29T20:48:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: _finish_auth 0 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 1 -- 172.21.15.64:0/1726290374 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f4031830 con 0x7fc50406d0e0 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 1 -- 172.21.15.64:0/1726290374 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4031c00 con 0x7fc50406d0e0 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc509b3d640 5 monclient: authenticate success, global_id 22159 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc509b3d640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc509b3d640 4 set_mon_vals no callback set 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.315+0000 7fc502ffd640 10 monclient: dump: 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.318 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 10 monclient: shutdown 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 -- 172.21.15.64:0/1726290374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc50406d0e0 msgr2=0x7fc5040aae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 --2- 172.21.15.64:0/1726290374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc50406d0e0 0x7fc5040aae60 secure :-1 s=READY pgs=5628 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40097a0 tx=0x7fc4f402b8d0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 -- 172.21.15.64:0/1726290374 shutdown_connections 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 --2- 172.21.15.64:0/1726290374 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc504146ec0 0x7fc50414dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 --2- 172.21.15.64:0/1726290374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc50406d0e0 0x7fc5040aae60 unknown :-1 s=CLOSED pgs=5628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 -- 172.21.15.64:0/1726290374 >> 172.21.15.64:0/1726290374 conn(0x7fc5040b9240 msgr2=0x7fc5040b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 -- 172.21.15.64:0/1726290374 shutdown_connections 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.316+0000 7fc509b3d640 1 -- 172.21.15.64:0/1726290374 wait complete. 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 10 monclient: build_initial_monmap 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 10 monclient: monmap: 2024-04-29T20:48:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 1 librados: starting msgr at 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 1 librados: starting objecter 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 1 Processor -- start 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 1 -- start start 2024-04-29T20:48:26.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 1 librados: setting wanted keys 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 1 librados: calling monclient init 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.317+0000 7fc509b3d640 10 monclient: init 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc50406d0e0 0x7fc5040ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fc50406d0e0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc504146a50 0x7fc504143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fc504146a50 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc504146ec0 0x7fc5041dbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fc504146ec0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc5040b56e0 con 0x7fc504146ec0 2024-04-29T20:48:26.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc5040b5560 con 0x7fc504146a50 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fc504140b10 con 0x7fc50406d0e0 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc509b3d640 10 monclient(hunting): authenticate will time out at 3055.477823s 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc50406d0e0 0x7fc5040ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc503fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc504146ec0 0x7fc5041dbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.318+0000 7fc4fbfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc504146a50 0x7fc504143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc504146a50 0x7fc504143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56586/0 (socket says 172.21.15.64:56586) 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 1 -- 172.21.15.64:0/4255536168 learned_addr learned my addr 172.21.15.64:0/4255536168 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): get_auth_request con 0x7fc504146a50 auth_method 0 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): get_auth_request con 0x7fc50406d0e0 auth_method 0 2024-04-29T20:48:26.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): get_auth_request con 0x7fc504146ec0 auth_method 0 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc4fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc5037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.319+0000 7fc503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 10 monclient(hunting): handle_auth_done global_id 22162 payload 824 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 1 -- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc50406d0e0 msgr2=0x7fc5040ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fc50406d0e0 0x7fc5040ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 1 -- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc504146ec0 msgr2=0x7fc5041dbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc504146ec0 0x7fc5041dbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 10 monclient: _finish_hunting 0 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 1 monclient: found mon.noname-b 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.322+0000 7fc4fbfff640 1 -- 172.21.15.64:0/4255536168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5040ae890 con 0x7fc504146a50 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.324+0000 7fc4fbfff640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc504146a50 0x7fc504143480 secure :-1 s=READY pgs=5629 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00c400 tx=0x7fc4ec00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.324+0000 7fc500ff9640 10 client.?.objecter ms_handle_connect 0x7fc504146a50 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.324+0000 7fc500ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec017070 con 0x7fc504146a50 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: dump: 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.329 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: _finish_auth 0 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ec005ce0 con 0x7fc504146a50 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0072f0 con 0x7fc504146a50 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc500ff9640 10 monclient: dump: 2024-04-29T20:48:26.330 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc509b3d640 5 monclient: authenticate success, global_id 22162 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 monclient: _renew_subs 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5040af650 con 0x7fc504146a50 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.327+0000 7fc5027fc640 4 set_mon_vals no callback set 2024-04-29T20:48:26.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 client.22162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 monclient: _renew_subs 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5041dc090 con 0x7fc504146a50 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 1 librados: init done 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 client.22162.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 15 client.22162.objecter _session_command_op_assign -1 1 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 20 client.22162.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 client.22162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:26.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.328+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc5040aef90 con 0x7fc504146a50 2024-04-29T20:48:26.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.347+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc4ec01a070 con 0x7fc504146a50 2024-04-29T20:48:26.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.347+0000 7fc500ff9640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc4e406e280 0x7fc4e4070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.348+0000 7fc5037fe640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc4e406e280 0x7fc4e4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.348+0000 7fc5037fe640 10 monclient: get_auth_request con 0x7fc4e406e280 auth_method 0 2024-04-29T20:48:26.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.348+0000 7fc500ff9640 10 client.22162.objecter ms_handle_connect 0x7fc4e406e280 2024-04-29T20:48:26.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.348+0000 7fc5037fe640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc4e406e280 0x7fc4e4070740 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4036000 tx=0x7fc4f4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7fc4ec0574a0 con 0x7fc504146a50 2024-04-29T20:48:26.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 10 client.22162.objecter ms_dispatch 0x7fc504000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:26.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 3 client.22162.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 3 client.22162.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 10 client.22162.objecter checking command 1 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7fc4e4075ef0 0x7fc4e4078390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 20 client.22162.objecter _get_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 15 client.22162.objecter _session_command_op_remove -1 1 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 20 client.22162.objecter _get_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 20 client.22162.objecter get_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 15 client.22162.objecter _session_command_op_assign 2 1 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 4 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 10 client.22162.objecter _send_command 1 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 --> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4e4078a60 con 0x7fc4e4075ef0 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.351+0000 7fc500ff9640 20 client.22162.objecter dump_active .. 0 homeless 2024-04-29T20:48:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.352+0000 7fc503fff640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7fc4e4075ef0 0x7fc4e4078390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.352+0000 7fc503fff640 10 monclient: get_auth_request con 0x7fc4e4075ef0 auth_method 0 2024-04-29T20:48:26.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.353+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7fc4ec0556a0 con 0x7fc504146a50 2024-04-29T20:48:26.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.353+0000 7fc500ff9640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:26.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.355+0000 7fc500ff9640 10 client.22162.objecter ms_handle_connect 0x7fc4e4075ef0 2024-04-29T20:48:26.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.355+0000 7fc503fff640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7fc4e4075ef0 0x7fc4e4078390 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.379+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== osd.2 v2:172.21.15.64:6800/2373308255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4e4078a60 con 0x7fc4e4075ef0 2024-04-29T20:48:26.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.379+0000 7fc500ff9640 10 client.22162.objecter ms_dispatch 0x7fc504000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:26.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.379+0000 7fc500ff9640 10 client.22162.objecter _finish_command 1 = system:0 2024-04-29T20:48:26.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.379+0000 7fc500ff9640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 4 2024-04-29T20:48:26.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.379+0000 7fc500ff9640 15 client.22162.objecter _session_command_op_remove 2 1 2024-04-29T20:48:26.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 10 client.22162.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 15 client.22162.objecter _session_command_op_assign -1 2 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 20 client.22162.objecter _get_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 20 client.22162.objecter _get_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 15 client.22162.objecter _session_command_op_remove -1 2 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 20 client.22162.objecter get_session s=0x7fc4e4075820 osd=2 3 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 15 client.22162.objecter _session_command_op_assign 2 2 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 4 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 10 client.22162.objecter _send_command 2 2024-04-29T20:48:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.393+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 --> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc50404fce0 con 0x7fc4e4075ef0 2024-04-29T20:48:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.406+0000 7fc500ff9640 1 -- 172.21.15.64:0/4255536168 <== osd.2 v2:172.21.15.64:6800/2373308255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc50404fce0 con 0x7fc4e4075ef0 2024-04-29T20:48:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.406+0000 7fc500ff9640 10 client.22162.objecter ms_dispatch 0x7fc504000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.406+0000 7fc500ff9640 10 client.22162.objecter _finish_command 2 = system:0 2024-04-29T20:48:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.406+0000 7fc500ff9640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 4 2024-04-29T20:48:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.406+0000 7fc500ff9640 15 client.22162.objecter _session_command_op_remove 2 2 2024-04-29T20:48:26.407 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:26.407 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:26.407 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 10 librados: watch_flush enter 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 10 librados: watch_flush exit 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 10 client.22162.objecter close_session for osd.2 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7fc4e4075ef0 msgr2=0x7fc4e4078390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.64:6800/2373308255,v1:172.21.15.64:6801/2373308255] conn(0x7fc4e4075ef0 0x7fc4e4078390 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 20 client.22162.objecter put_session s=0x7fc4e4075820 osd=2 1 2024-04-29T20:48:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 20 client.22162.objecter shutdown clearing up homeless session... 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 10 client.22162.objecter successfully canceled tick 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc4e406e280 msgr2=0x7fc4e4070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fc4e406e280 0x7fc4e4070740 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4036000 tx=0x7fc4f4008030 comp rx=0 tx=0).stop 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 10 monclient: shutdown 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc504146a50 msgr2=0x7fc504143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 --2- 172.21.15.64:0/4255536168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc504146a50 0x7fc504143480 secure :-1 s=READY pgs=5629 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00c400 tx=0x7fc4ec00c8c0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc503fff640 1 -- 172.21.15.64:0/4255536168 reap_dead start 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 shutdown_connections 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 >> 172.21.15.64:0/4255536168 conn(0x7fc5040b9240 msgr2=0x7fc50413ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 shutdown_connections 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 -- 172.21.15.64:0/4255536168 wait complete. 2024-04-29T20:48:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.407+0000 7fc509b3d640 1 librados: shutdown 2024-04-29T20:48:26.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.523+0000 7ff4b7fc7640 10 monclient: get_monmap_and_config 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.523+0000 7ff4b7fc7640 10 monclient: build_initial_monmap 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.523+0000 7ff4b7fc7640 10 monclient: monmap: 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:26.524455+0000 2024-04-29T20:48:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:26.524455+0000 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:26.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.524+0000 7ff4b7fc7640 1 Processor -- start 2024-04-29T20:48:26.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.524+0000 7ff4b7fc7640 1 -- start start 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.524+0000 7ff4b7fc7640 10 monclient: init 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.524+0000 7ff4b7fc7640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.524+0000 7ff4b7fc7640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.524+0000 7ff4b7fc7640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b22a0 0x7ff4b0146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b00b22a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff4b00b2c20 0x7ff4b00b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b2c20 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4b00aa6a0 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b00ac1f0 con 0x7ff4b00aa6a0 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b00b33d0 con 0x7ff4b00b2c20 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b00ac4d0 con 0x7ff4b00b22a0 2024-04-29T20:48:26.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b7fc7640 10 monclient(hunting): authenticate will time out at 3055.684642s 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.525+0000 7ff4b5562640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff4b00b2c20 0x7ff4b00b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7ff4b00b2c20 0x7ff4b00b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36820/0 (socket says 172.21.15.64:36820) 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 1 -- 172.21.15.64:0/240451163 learned_addr learned my addr 172.21.15.64:0/240451163 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): get_auth_request con 0x7ff4b00b2c20 auth_method 0 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.526+0000 7ff4b5562640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 10 monclient(hunting): handle_auth_done global_id 14652 payload 274 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 1 -- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b22a0 msgr2=0x7ff4b0146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 1 --2- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b22a0 0x7ff4b0146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 1 -- 172.21.15.64:0/240451163 >> v1:172.21.15.167:6789/0 conn(0x7ff4b00aa6a0 legacy=0x7ff4b00b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 10 monclient: _finish_hunting 0 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 1 monclient: found mon.noname-a 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:26.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:48:26.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.528+0000 7ff4b5562640 1 -- 172.21.15.64:0/240451163 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b00b3040 con 0x7ff4b00b2c20 2024-04-29T20:48:26.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.530+0000 7ff4b5562640 1 --2- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b2c20 0x7ff4b00b5b30 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009870 tx=0x7ff4a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921984d64a5c8ae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 1 -- 172.21.15.64:0/240451163 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0035020 con 0x7ff4b00b2c20 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: dump: 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: _finish_auth 0 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: _check_auth_tickets 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 1 -- 172.21.15.64:0/240451163 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a0031640 con 0x7ff4b00b2c20 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 1 -- 172.21.15.64:0/240451163 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a002d040 con 0x7ff4b00b2c20 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b7fc7640 5 monclient: authenticate success, global_id 14652 2024-04-29T20:48:26.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b7fc7640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b7fc7640 4 set_mon_vals no callback set 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.534+0000 7ff4b4d61640 10 monclient: dump: 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.536 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 10 monclient: shutdown 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b2c20 msgr2=0x7ff4b00b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b2c20 0x7ff4b00b5b30 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009870 tx=0x7ff4a002b8f0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/240451163 shutdown_connections 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b2c20 0x7ff4b00b5b30 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/240451163 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b22a0 0x7ff4b0146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/240451163 >> 172.21.15.64:0/240451163 conn(0x7ff4b006c940 msgr2=0x7ff4b006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/240451163 shutdown_connections 2024-04-29T20:48:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.535+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/240451163 wait complete. 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.536+0000 7ff4b7fc7640 10 monclient: build_initial_monmap 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.536+0000 7ff4b7fc7640 10 monclient: monmap: 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 1 librados: starting msgr at 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 1 librados: starting objecter 2024-04-29T20:48:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 1 Processor -- start 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 1 -- start start 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 1 librados: setting wanted keys 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 1 librados: calling monclient init 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.537+0000 7ff4b7fc7640 10 monclient: init 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff4b00aa6a0 0x7ff4b013e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b00aa6a0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 0x7ff4b00ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b22a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b2c20 0x7ff4b013cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b00b2c20 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b00b2800 con 0x7ff4b00b22a0 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b00b2680 con 0x7ff4b00aa6a0 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b00aab30 con 0x7ff4b00b2c20 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b7fc7640 10 monclient(hunting): authenticate will time out at 3055.697879s 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.538+0000 7ff4b5562640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 0x7ff4b00ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 0x7ff4b00ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36836/0 (socket says 172.21.15.64:36836) 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 1 -- 172.21.15.64:0/4251104535 learned_addr learned my addr 172.21.15.64:0/4251104535 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): get_auth_request con 0x7ff4b00b22a0 auth_method 0 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:26.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.539+0000 7ff4b5562640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:26.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 10 monclient(hunting): handle_auth_done global_id 14655 payload 824 2024-04-29T20:48:26.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 1 -- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b2c20 msgr2=0x7ff4b013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b2c20 0x7ff4b013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 1 -- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff4b00aa6a0 msgr2=0x7ff4b013e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff4b00aa6a0 0x7ff4b013e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 10 monclient: _finish_hunting 0 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 1 monclient: found mon.noname-a 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.542+0000 7ff4b5562640 1 -- 172.21.15.64:0/4251104535 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b00aecd0 con 0x7ff4b00b22a0 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.544+0000 7ff4b5562640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 0x7ff4b00ac840 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7ff4a002b930 tx=0x7ff4a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.544+0000 7ff4a67fc640 10 client.?.objecter ms_handle_connect 0x7ff4b00b22a0 2024-04-29T20:48:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.544+0000 7ff4a67fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:26.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a003ac60 con 0x7ff4b00b22a0 2024-04-29T20:48:26.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: dump: 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: _finish_auth 0 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: _check_auth_tickets 2024-04-29T20:48:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a003ae20 con 0x7ff4b00b22a0 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0044630 con 0x7ff4b00b22a0 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 5 monclient: authenticate success, global_id 14655 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 10 monclient: _renew_subs 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b00ace60 con 0x7ff4b00b22a0 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a7fff640 4 set_mon_vals no callback set 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4a67fc640 10 monclient: dump: 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 10 client.14655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 10 monclient: _renew_subs 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b00adc00 con 0x7ff4b00b22a0 2024-04-29T20:48:26.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.548+0000 7ff4b7fc7640 1 librados: init done 2024-04-29T20:48:26.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.549+0000 7ff4b7fc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:26.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.549+0000 7ff4b7fc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:26.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.549+0000 7ff4b7fc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.64:3300/0 2024-04-29T20:48:26.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.549+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b004fce0 con 0x7ff4b00b22a0 2024-04-29T20:48:26.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.570+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff4a0038020 con 0x7ff4b00b22a0 2024-04-29T20:48:26.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.571+0000 7ff4a67fc640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff49406e2d0 0x7ff494070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:26.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.571+0000 7ff4b5d63640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff49406e2d0 0x7ff494070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:26.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.571+0000 7ff4b5d63640 10 monclient: get_auth_request con 0x7ff49406e2d0 auth_method 0 2024-04-29T20:48:26.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.572+0000 7ff4a67fc640 10 client.14655.objecter ms_handle_connect 0x7ff49406e2d0 2024-04-29T20:48:26.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.572+0000 7ff4b5d63640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff49406e2d0 0x7ff494070790 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009710 tx=0x7ff4ac006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.574+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7ff4a0079e40 con 0x7ff4b00b22a0 2024-04-29T20:48:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.574+0000 7ff4a67fc640 10 client.14655.objecter ms_dispatch 0x7ff4b0000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.574+0000 7ff4a67fc640 3 client.14655.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.574+0000 7ff4a67fc640 3 client.14655.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.574+0000 7ff4a67fc640 20 client.14655.objecter dump_active .. 0 homeless 2024-04-29T20:48:26.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.835+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4a003f070 con 0x7ff4b00b22a0 2024-04-29T20:48:26.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.835+0000 7ff4a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:26.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:26.835+0000 7ff4a67fc640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:27.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.036+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ff4b0071f90 con 0x7ff49406e2d0 2024-04-29T20:48:27.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.037+0000 7ff4a67fc640 1 -- 172.21.15.64:0/4251104535 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff4b0071f90 con 0x7ff49406e2d0 2024-04-29T20:48:27.037 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 2 to deep-scrub 2024-04-29T20:48:27.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 10 librados: watch_flush enter 2024-04-29T20:48:27.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 10 librados: watch_flush exit 2024-04-29T20:48:27.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 20 client.14655.objecter shutdown clearing up homeless session... 2024-04-29T20:48:27.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 10 client.14655.objecter successfully canceled tick 2024-04-29T20:48:27.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff49406e2d0 msgr2=0x7ff494070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff49406e2d0 0x7ff494070790 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009710 tx=0x7ff4ac006c90 comp rx=0 tx=0).stop 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 10 monclient: shutdown 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 msgr2=0x7ff4b00ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 0x7ff4b00ac840 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7ff4a002b930 tx=0x7ff4a00046c0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 shutdown_connections 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7ff49406e2d0 0x7ff494070790 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7ff4b00b2c20 0x7ff4b013cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4b00b22a0 0x7ff4b00ac840 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 --2- 172.21.15.64:0/4251104535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff4b00aa6a0 0x7ff4b013e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 >> 172.21.15.64:0/4251104535 conn(0x7ff4b006c940 msgr2=0x7ff4b01404d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 shutdown_connections 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 -- 172.21.15.64:0/4251104535 wait complete. 2024-04-29T20:48:27.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.040+0000 7ff4b7fc7640 1 librados: shutdown 2024-04-29T20:48:27.053 INFO:tasks.ceph:Scrubbing osd.3 2024-04-29T20:48:27.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:27.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.164+0000 7f5b5cc6d640 10 monclient: get_monmap_and_config 2024-04-29T20:48:27.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.164+0000 7f5b5cc6d640 10 monclient: build_initial_monmap 2024-04-29T20:48:27.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.164+0000 7f5b5cc6d640 10 monclient: monmap: 2024-04-29T20:48:27.164 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:27.164 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.164 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:27.165387+0000 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:27.165387+0000 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:27.165 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.165+0000 7f5b5cc6d640 1 Processor -- start 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.165+0000 7f5b5cc6d640 1 -- start start 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.165+0000 7f5b5cc6d640 10 monclient: init 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b580b5b90 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f5b580b5f70 0x7f5b58146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b580b5f70 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b580aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580aabc0 con 0x7f5b580b5b90 2024-04-29T20:48:27.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580aa8f0 con 0x7f5b580aa460 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f5b580b2530 con 0x7f5b580b5f70 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5cc6d640 10 monclient(hunting): authenticate will time out at 3056.325820s 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5b580b5b90 0x7f5b580b2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60504/0 (socket says 172.21.15.64:60504) 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.166+0000 7f5b5659c640 1 -- 172.21.15.64:0/3699627615 learned_addr learned my addr 172.21.15.64:0/3699627615 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.167+0000 7f5b55d9b640 1 --2- 172.21.15.64:0/3699627615 >> v2:172.21.15.167:3300/0 conn(0x7f5b580b5f70 0x7f5b58146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.167+0000 7f5b55d9b640 10 monclient(hunting): get_auth_request con 0x7f5b580b5f70 auth_method 0 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.167+0000 7f5b55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.167+0000 7f5b55d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.167+0000 7f5b55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.168+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.168+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.168+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_done global_id 22165 payload 274 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 1 -- 172.21.15.64:0/3699627615 >> v1:172.21.15.167:6789/0 conn(0x7f5b580aa460 legacy=0x7f5b580b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 1 -- 172.21.15.64:0/3699627615 >> v1:172.21.15.64:6789/0 conn(0x7f5b580b5b90 legacy=0x7f5b580b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 10 monclient: _finish_hunting 0 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 1 monclient: found mon.noname-c 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.170+0000 7f5b55d9b640 1 -- 172.21.15.64:0/3699627615 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b580b26b0 con 0x7f5b580b5f70 2024-04-29T20:48:27.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.174+0000 7f5b55d9b640 1 --2- 172.21.15.64:0/3699627615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5f70 0x7f5b58146900 secure :-1 s=READY pgs=5631 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097a0 tx=0x7f5b4c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e311a02fa840f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 1 -- 172.21.15.64:0/3699627615 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c034070 con 0x7f5b580b5f70 2024-04-29T20:48:27.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:27.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: dump: 2024-04-29T20:48:27.177 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: _finish_auth 0 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: _check_auth_tickets 2024-04-29T20:48:27.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 1 -- 172.21.15.64:0/3699627615 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4c030610 con 0x7f5b580b5f70 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 1 -- 172.21.15.64:0/3699627615 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c0309e0 con 0x7f5b580b5f70 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5cc6d640 5 monclient: authenticate success, global_id 22165 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5cc6d640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5cc6d640 4 set_mon_vals no callback set 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.177+0000 7f5b5559a640 10 monclient: dump: 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.179 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.180 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.180 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.180 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.180 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.180 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 10 monclient: shutdown 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/3699627615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5f70 msgr2=0x7f5b58146900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 --2- 172.21.15.64:0/3699627615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5f70 0x7f5b58146900 secure :-1 s=READY pgs=5631 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0097a0 tx=0x7f5b4c02b960 comp rx=0 tx=0).stop 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/3699627615 shutdown_connections 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 --2- 172.21.15.64:0/3699627615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5f70 0x7f5b58146900 unknown :-1 s=CLOSED pgs=5631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/3699627615 >> 172.21.15.64:0/3699627615 conn(0x7f5b5806c9d0 msgr2=0x7f5b5806cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/3699627615 shutdown_connections 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.178+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/3699627615 wait complete. 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.179+0000 7f5b5cc6d640 10 monclient: build_initial_monmap 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.179+0000 7f5b5cc6d640 10 monclient: monmap: 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:27.183 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.179+0000 7f5b5cc6d640 1 librados: starting msgr at 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.179+0000 7f5b5cc6d640 1 librados: starting objecter 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 Processor -- start 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 -- start start 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 librados: setting wanted keys 2024-04-29T20:48:27.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 librados: calling monclient init 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient: init 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5b580aa460 0x7f5b580acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b580aa460 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5b90 0x7f5b580ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b580b5b90 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b580ad9c0 0x7f5b58139990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b580ad9c0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5814ee20 con 0x7f5b580ad9c0 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5814eca0 con 0x7f5b580b5b90 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5b5814efa0 con 0x7f5b580aa460 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.180+0000 7f5b5cc6d640 10 monclient(hunting): authenticate will time out at 3056.340267s 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5b580aa460 0x7f5b580acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b55d9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5b90 0x7f5b580ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5b580aa460 0x7f5b580acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45680/0 (socket says 172.21.15.64:45680) 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b55d9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5b90 0x7f5b580ad480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56618/0 (socket says 172.21.15.64:56618) 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 1 -- 172.21.15.64:0/2171741741 learned_addr learned my addr 172.21.15.64:0/2171741741 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 10 monclient(hunting): get_auth_request con 0x7f5b580aa460 auth_method 0 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b55d9b640 10 monclient(hunting): get_auth_request con 0x7f5b580b5b90 auth_method 0 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b55d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.181+0000 7f5b55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.182+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.182+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.182+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.182+0000 7f5b5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.182+0000 7f5b5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.182+0000 7f5b5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 10 monclient(hunting): handle_auth_done global_id 22168 payload 824 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 1 -- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5b580aa460 msgr2=0x7f5b580acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5b580aa460 0x7f5b580acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 1 -- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b580ad9c0 msgr2=0x7f5b58139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b580ad9c0 0x7f5b58139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 10 monclient: _finish_hunting 0 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 1 monclient: found mon.noname-b 2024-04-29T20:48:27.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:27.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.184+0000 7f5b55d9b640 1 -- 172.21.15.64:0/2171741741 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b580aef00 con 0x7f5b580b5b90 2024-04-29T20:48:27.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.187+0000 7f5b55d9b640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5b90 0x7f5b580ad480 secure :-1 s=READY pgs=5632 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c033000 tx=0x7f5b4c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.187+0000 7f5b46ffd640 10 client.?.objecter ms_handle_connect 0x7f5b580b5b90 2024-04-29T20:48:27.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.187+0000 7f5b46ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:27.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c034040 con 0x7f5b580b5b90 2024-04-29T20:48:27.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:27.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 10 monclient: dump: 2024-04-29T20:48:27.191 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.191 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 10 monclient: _finish_auth 0 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 5 monclient: authenticate success, global_id 22168 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 10 monclient: _renew_subs 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b580af1e0 con 0x7f5b580b5b90 2024-04-29T20:48:27.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 10 client.22168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 10 monclient: _renew_subs 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b580af600 con 0x7f5b580b5b90 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b5cc6d640 1 librados: init done 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4c0304e0 con 0x7f5b580b5b90 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b46ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.190+0000 7f5b54d99640 4 set_mon_vals no callback set 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c04c3f0 con 0x7f5b580b5b90 2024-04-29T20:48:27.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b46ffd640 10 monclient: dump: 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b5cc6d640 10 client.22168.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:27.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b5cc6d640 15 client.22168.objecter _session_command_op_assign -1 1 2024-04-29T20:48:27.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b5cc6d640 20 client.22168.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:27.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b5cc6d640 10 client.22168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:27.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b5cc6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.191+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b3c002910 con 0x7f5b580b5b90 2024-04-29T20:48:27.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.214+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b4c04b060 con 0x7f5b580b5b90 2024-04-29T20:48:27.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.214+0000 7f5b46ffd640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5b3806e2b0 0x7f5b38070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.214+0000 7f5b5659c640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5b3806e2b0 0x7f5b38070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.214+0000 7f5b5659c640 10 monclient: get_auth_request con 0x7f5b3806e2b0 auth_method 0 2024-04-29T20:48:27.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.215+0000 7f5b46ffd640 10 client.22168.objecter ms_handle_connect 0x7f5b3806e2b0 2024-04-29T20:48:27.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.215+0000 7f5b5659c640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5b3806e2b0 0x7f5b38070770 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005880 tx=0x7f5b48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f5b4c079830 con 0x7f5b580b5b90 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 10 client.22168.objecter ms_dispatch 0x7f5b58000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 3 client.22168.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 3 client.22168.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 10 client.22168.objecter checking command 1 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f5b38075fb0 0x7f5b38078450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 20 client.22168.objecter _get_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 15 client.22168.objecter _session_command_op_remove -1 1 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 20 client.22168.objecter _get_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 20 client.22168.objecter get_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 15 client.22168.objecter _session_command_op_assign 3 1 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 4 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 10 client.22168.objecter _send_command 1 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 --> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b4c04bec0 con 0x7f5b38075fb0 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.217+0000 7f5b46ffd640 20 client.22168.objecter dump_active .. 0 homeless 2024-04-29T20:48:27.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.219+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7f5b4c077a30 con 0x7f5b580b5b90 2024-04-29T20:48:27.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.219+0000 7f5b46ffd640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:27.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.222+0000 7f5b56d9d640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f5b38075fb0 0x7f5b38078450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.223+0000 7f5b56d9d640 10 monclient: get_auth_request con 0x7f5b38075fb0 auth_method 0 2024-04-29T20:48:27.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.225+0000 7f5b46ffd640 10 client.22168.objecter ms_handle_connect 0x7f5b38075fb0 2024-04-29T20:48:27.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.225+0000 7f5b56d9d640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f5b38075fb0 0x7f5b38078450 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.268+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== osd.3 v2:172.21.15.64:6824/471606907 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b4c04bec0 con 0x7f5b38075fb0 2024-04-29T20:48:27.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.268+0000 7f5b46ffd640 10 client.22168.objecter ms_dispatch 0x7f5b58000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:27.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.268+0000 7f5b46ffd640 10 client.22168.objecter _finish_command 1 = system:0 2024-04-29T20:48:27.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.268+0000 7f5b46ffd640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 4 2024-04-29T20:48:27.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.268+0000 7f5b46ffd640 15 client.22168.objecter _session_command_op_remove 3 1 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 10 client.22168.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 15 client.22168.objecter _session_command_op_assign -1 2 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 20 client.22168.objecter _get_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 20 client.22168.objecter _get_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 15 client.22168.objecter _session_command_op_remove -1 2 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 20 client.22168.objecter get_session s=0x7f5b380758e0 osd=3 3 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 15 client.22168.objecter _session_command_op_assign 3 2 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 4 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 10 client.22168.objecter _send_command 2 2024-04-29T20:48:27.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.284+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 --> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5b3c0012d0 con 0x7f5b38075fb0 2024-04-29T20:48:27.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.300+0000 7f5b46ffd640 1 -- 172.21.15.64:0/2171741741 <== osd.3 v2:172.21.15.64:6824/471606907 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5b3c0012d0 con 0x7f5b38075fb0 2024-04-29T20:48:27.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.300+0000 7f5b46ffd640 10 client.22168.objecter ms_dispatch 0x7f5b58000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:27.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.300+0000 7f5b46ffd640 10 client.22168.objecter _finish_command 2 = system:0 2024-04-29T20:48:27.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.300+0000 7f5b46ffd640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 4 2024-04-29T20:48:27.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.300+0000 7f5b46ffd640 15 client.22168.objecter _session_command_op_remove 3 2 2024-04-29T20:48:27.301 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:27.301 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:27.301 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:27.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 10 librados: watch_flush enter 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 10 librados: watch_flush exit 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 10 client.22168.objecter close_session for osd.3 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f5b38075fb0 msgr2=0x7f5b38078450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.64:6824/471606907,v1:172.21.15.64:6825/471606907] conn(0x7f5b38075fb0 0x7f5b38078450 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 20 client.22168.objecter put_session s=0x7f5b380758e0 osd=3 1 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 20 client.22168.objecter shutdown clearing up homeless session... 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 10 client.22168.objecter successfully canceled tick 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5b3806e2b0 msgr2=0x7f5b38070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5b3806e2b0 0x7f5b38070770 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005880 tx=0x7f5b48009040 comp rx=0 tx=0).stop 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 10 monclient: shutdown 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5b90 msgr2=0x7f5b580ad480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 --2- 172.21.15.64:0/2171741741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b580b5b90 0x7f5b580ad480 secure :-1 s=READY pgs=5632 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c033000 tx=0x7f5b4c0046c0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b56d9d640 1 -- 172.21.15.64:0/2171741741 reap_dead start 2024-04-29T20:48:27.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 shutdown_connections 2024-04-29T20:48:27.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 >> 172.21.15.64:0/2171741741 conn(0x7f5b5806c9d0 msgr2=0x7f5b5806a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:27.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 shutdown_connections 2024-04-29T20:48:27.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 -- 172.21.15.64:0/2171741741 wait complete. 2024-04-29T20:48:27.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.301+0000 7f5b5cc6d640 1 librados: shutdown 2024-04-29T20:48:27.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-29T20:48:27.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.412+0000 7f36ab711640 10 monclient: get_monmap_and_config 2024-04-29T20:48:27.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.412+0000 7f36ab711640 10 monclient: build_initial_monmap 2024-04-29T20:48:27.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.412+0000 7f36ab711640 10 monclient: monmap: 2024-04-29T20:48:27.412 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:27.412 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:27.413497+0000 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:27.413497+0000 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:27.413 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.413+0000 7f36ab711640 1 Processor -- start 2024-04-29T20:48:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.413+0000 7f36ab711640 1 -- start start 2024-04-29T20:48:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.413+0000 7f36ab711640 10 monclient: init 2024-04-29T20:48:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.413+0000 7f36ab711640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36a40b23a0 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f36a406f740 0x7f36a41402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): picked mon.noname-c con 0x7f36a406f740 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f36a4140820 0x7f36a414dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): picked mon.noname-a con 0x7f36a4140820 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36a40709f0 con 0x7f36a40b23a0 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f36a4147080 con 0x7f36a4140820 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f36a406fbd0 con 0x7f36a406f740 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36ab711640 10 monclient(hunting): authenticate will time out at 3056.573701s 2024-04-29T20:48:27.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36a9cae640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f36a4140820 0x7f36a414dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.414+0000 7f36a8cac640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f36a406f740 0x7f36a41402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f36a4140820 0x7f36a414dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36864/0 (socket says 172.21.15.64:36864) 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 1 -- 172.21.15.64:0/3833155418 learned_addr learned my addr 172.21.15.64:0/3833155418 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): get_auth_request con 0x7f36a4140820 auth_method 0 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a8cac640 10 monclient(hunting): get_auth_request con 0x7f36a406f740 auth_method 0 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a8cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a8cac640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a8cac640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.415+0000 7f36a9cae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.416+0000 7f36a8cac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.416+0000 7f36a8cac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.416+0000 7f36a8cac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 10 monclient(hunting): handle_auth_done global_id 14661 payload 274 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 1 -- 172.21.15.64:0/3833155418 >> v2:172.21.15.167:3300/0 conn(0x7f36a406f740 msgr2=0x7f36a41402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 1 --2- 172.21.15.64:0/3833155418 >> v2:172.21.15.167:3300/0 conn(0x7f36a406f740 0x7f36a41402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 1 -- 172.21.15.64:0/3833155418 >> v1:172.21.15.167:6789/0 conn(0x7f36a40b23a0 legacy=0x7f36a4146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 10 monclient: _finish_hunting 0 2024-04-29T20:48:27.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 1 monclient: found mon.noname-a 2024-04-29T20:48:27.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:27.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:48:27.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.417+0000 7f36a9cae640 1 -- 172.21.15.64:0/3833155418 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a4070c60 con 0x7f36a4140820 2024-04-29T20:48:27.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.419+0000 7f36a9cae640 1 --2- 172.21.15.64:0/3833155418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a4140820 0x7f36a414dd00 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f36a406e2e0 tx=0x7f369c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ab4dfef86eb099 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 1 -- 172.21.15.64:0/3833155418 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c02c9f0 con 0x7f36a4140820 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: dump: 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.423 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: _finish_auth 0 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: _check_auth_tickets 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 1 -- 172.21.15.64:0/3833155418 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369c02cb90 con 0x7f36a4140820 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f36ab711640 5 monclient: authenticate success, global_id 14661 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f36ab711640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f36ab711640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f36ab711640 4 set_mon_vals no callback set 2024-04-29T20:48:27.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 1 -- 172.21.15.64:0/3833155418 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c0343f0 con 0x7f36a4140820 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.423+0000 7f369bfff640 10 monclient: dump: 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.425 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 10 monclient: shutdown 2024-04-29T20:48:27.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:27.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 -- 172.21.15.64:0/3833155418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a4140820 msgr2=0x7f36a414dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 --2- 172.21.15.64:0/3833155418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a4140820 0x7f36a414dd00 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f36a406e2e0 tx=0x7f369c02be50 comp rx=0 tx=0).stop 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 -- 172.21.15.64:0/3833155418 shutdown_connections 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 --2- 172.21.15.64:0/3833155418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a4140820 0x7f36a414dd00 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 --2- 172.21.15.64:0/3833155418 >> v2:172.21.15.167:3300/0 conn(0x7f36a406f740 0x7f36a41402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 -- 172.21.15.64:0/3833155418 >> 172.21.15.64:0/3833155418 conn(0x7f36a40b54f0 msgr2=0x7f36a40b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 -- 172.21.15.64:0/3833155418 shutdown_connections 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 1 -- 172.21.15.64:0/3833155418 wait complete. 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 10 monclient: build_initial_monmap 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.424+0000 7f36ab711640 10 monclient: monmap: 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:27.429 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 librados: starting msgr at 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 librados: starting objecter 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 Processor -- start 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 -- start start 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 librados: setting wanted keys 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 librados: calling monclient init 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient: init 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:27.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36a406f740 0x7f36a4145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): picked mon.noname-c con 0x7f36a406f740 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a40b23a0 0x7f36a4142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): picked mon.noname-a con 0x7f36a40b23a0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36a4140820 0x7f36a41d8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): picked mon.noname-b con 0x7f36a4140820 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36a406eeb0 con 0x7f36a40b23a0 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f36a406f280 con 0x7f36a4140820 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f36a40af4d0 con 0x7f36a406f740 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:27.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.425+0000 7f36ab711640 10 monclient(hunting): authenticate will time out at 3056.585382s 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a40b23a0 0x7f36a4142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a94ad640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36a406f740 0x7f36a4145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a9cae640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36a4140820 0x7f36a41d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a40b23a0 0x7f36a4142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36878/0 (socket says 172.21.15.64:36878) 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 1 -- 172.21.15.64:0/1233221276 learned_addr learned my addr 172.21.15.64:0/1233221276 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 10 monclient(hunting): get_auth_request con 0x7f36a40b23a0 auth_method 0 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a8cac640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a94ad640 10 monclient(hunting): get_auth_request con 0x7f36a406f740 auth_method 0 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a94ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a94ad640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a94ad640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a9cae640 10 monclient(hunting): get_auth_request con 0x7f36a4140820 auth_method 0 2024-04-29T20:48:27.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a9cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a9cae640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.426+0000 7f36a9cae640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a8cac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a8cac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a8cac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a94ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a94ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a94ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a9cae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a9cae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:27.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.427+0000 7f36a9cae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:27.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 10 monclient(hunting): handle_auth_done global_id 22174 payload 824 2024-04-29T20:48:27.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 1 -- 172.21.15.64:0/1233221276 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36a406f740 msgr2=0x7f36a4145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36a406f740 0x7f36a4145a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 1 -- 172.21.15.64:0/1233221276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a40b23a0 msgr2=0x7f36a4142ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a40b23a0 0x7f36a4142ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 10 monclient: _finish_hunting 0 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 1 monclient: found mon.noname-b 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.430+0000 7f36a9cae640 1 -- 172.21.15.64:0/1233221276 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a41431e0 con 0x7f36a4140820 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.432+0000 7f36a9cae640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36a4140820 0x7f36a41d8290 secure :-1 s=READY pgs=5634 cs=0 l=1 rev1=1 crypto rx=0x7f36a40b5f80 tx=0x7f369c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.432+0000 7f3699ffb640 10 client.?.objecter ms_handle_connect 0x7f36a4140820 2024-04-29T20:48:27.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.432+0000 7f3699ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c044020 con 0x7f36a4140820 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: dump: 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: _finish_auth 0 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369c040470 con 0x7f36a4140820 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f36ab711640 5 monclient: authenticate success, global_id 22174 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c02c9f0 con 0x7f36a4140820 2024-04-29T20:48:27.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f3699ffb640 10 monclient: dump: 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f369b7fe640 4 set_mon_vals no callback set 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f36ab711640 10 monclient: _renew_subs 2024-04-29T20:48:27.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f36ab711640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.435+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a413a970 con 0x7f36a4140820 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.436+0000 7f36ab711640 10 client.22174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.436+0000 7f36ab711640 10 monclient: _renew_subs 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.436+0000 7f36ab711640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.436+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a41d89b0 con 0x7f36a4140820 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.436+0000 7f36ab711640 1 librados: init done 2024-04-29T20:48:27.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.437+0000 7f36ab711640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:27.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.437+0000 7f36ab711640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:27.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.437+0000 7f36ab711640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:27.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.437+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a4143410 con 0x7f36a4140820 2024-04-29T20:48:27.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.687+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f369c047020 con 0x7f36a4140820 2024-04-29T20:48:27.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.688+0000 7f3699ffb640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f368806e2d0 0x7f3688070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:27.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.688+0000 7f36a94ad640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f368806e2d0 0x7f3688070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:27.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.688+0000 7f36a94ad640 10 monclient: get_auth_request con 0x7f368806e2d0 auth_method 0 2024-04-29T20:48:27.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.688+0000 7f3699ffb640 10 client.22174.objecter ms_handle_connect 0x7f368806e2d0 2024-04-29T20:48:27.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.688+0000 7f36a94ad640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f368806e2d0 0x7f3688070790 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f36a0000f60 tx=0x7f36a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:27.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.691+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f369c07a770 con 0x7f36a4140820 2024-04-29T20:48:27.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.691+0000 7f3699ffb640 10 client.22174.objecter ms_dispatch 0x7f36a4000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:27.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.691+0000 7f3699ffb640 3 client.22174.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:27.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.691+0000 7f3699ffb640 3 client.22174.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:27.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.691+0000 7f3699ffb640 20 client.22174.objecter dump_active .. 0 homeless 2024-04-29T20:48:27.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.701+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f369c04e030 con 0x7f36a4140820 2024-04-29T20:48:27.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.701+0000 7f3699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:27.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.701+0000 7f3699ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.889+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f36a4143ae0 con 0x7f368806e2d0 2024-04-29T20:48:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.890+0000 7f3699ffb640 1 -- 172.21.15.64:0/1233221276 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f36a4143ae0 con 0x7f368806e2d0 2024-04-29T20:48:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 3 to deep-scrub 2024-04-29T20:48:27.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 10 librados: watch_flush enter 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 10 librados: watch_flush exit 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 20 client.22174.objecter shutdown clearing up homeless session... 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 10 client.22174.objecter successfully canceled tick 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f368806e2d0 msgr2=0x7f3688070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f368806e2d0 0x7f3688070790 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f36a0000f60 tx=0x7f36a0009040 comp rx=0 tx=0).stop 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 10 monclient: shutdown 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36a4140820 msgr2=0x7f36a41d8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36a4140820 0x7f36a41d8290 secure :-1 s=READY pgs=5634 cs=0 l=1 rev1=1 crypto rx=0x7f36a40b5f80 tx=0x7f369c004a40 comp rx=0 tx=0).stop 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 shutdown_connections 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f368806e2d0 0x7f3688070790 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36a4140820 0x7f36a41d8290 unknown :-1 s=CLOSED pgs=5634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a40b23a0 0x7f36a4142ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 --2- 172.21.15.64:0/1233221276 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f36a406f740 0x7f36a4145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 >> 172.21.15.64:0/1233221276 conn(0x7f36a40b54f0 msgr2=0x7f36a406a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:27.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 shutdown_connections 2024-04-29T20:48:27.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 -- 172.21.15.64:0/1233221276 wait complete. 2024-04-29T20:48:27.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:27.892+0000 7f36ab711640 1 librados: shutdown 2024-04-29T20:48:27.905 INFO:tasks.ceph:Scrubbing osd.4 2024-04-29T20:48:27.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.021+0000 7fba28225640 10 monclient: get_monmap_and_config 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.021+0000 7fba28225640 10 monclient: build_initial_monmap 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.021+0000 7fba28225640 10 monclient: monmap: 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:28.022226+0000 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:28.022226+0000 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.021 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:28.022 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:28.022 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:28.022 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:28.022 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:28.022 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.022+0000 7fba28225640 1 Processor -- start 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.022+0000 7fba28225640 1 -- start start 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.022+0000 7fba28225640 10 monclient: init 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fba2006cdb0 0x7fba201407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): picked mon.noname-a con 0x7fba2006cdb0 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba200b3270 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 0x7fba20147160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): picked mon.noname-b con 0x7fba20140d00 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba20141130 con 0x7fba200b3270 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7fba2006d190 con 0x7fba2006cdb0 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fba201476a0 con 0x7fba20140d00 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:28.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.023+0000 7fba28225640 10 monclient(hunting): authenticate will time out at 3057.183111s 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 0x7fba20147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 0x7fba20147160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45696/0 (socket says 172.21.15.64:45696) 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 1 -- 172.21.15.64:0/347565642 learned_addr learned my addr 172.21.15.64:0/347565642 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 10 monclient(hunting): get_auth_request con 0x7fba20140d00 auth_method 0 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.024+0000 7fba267c2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.025+0000 7fba267c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.025+0000 7fba267c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.025+0000 7fba267c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 10 monclient(hunting): handle_auth_done global_id 16382 payload 274 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 1 -- 172.21.15.64:0/347565642 >> v2:172.21.15.64:3300/0 conn(0x7fba2006cdb0 msgr2=0x7fba201407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 1 --2- 172.21.15.64:0/347565642 >> v2:172.21.15.64:3300/0 conn(0x7fba2006cdb0 0x7fba201407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 1 -- 172.21.15.64:0/347565642 >> v1:172.21.15.64:6789/0 conn(0x7fba200b3270 legacy=0x7fba20146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 10 monclient: _finish_hunting 0 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 1 monclient: found mon.noname-b 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:28.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.027+0000 7fba267c2640 1 -- 172.21.15.64:0/347565642 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba200abff0 con 0x7fba20140d00 2024-04-29T20:48:28.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.033+0000 7fba267c2640 1 --2- 172.21.15.64:0/347565642 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 0x7fba20147160 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fba18009e50 tx=0x7fba1802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d393b4c1be3572f server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 1 -- 172.21.15.64:0/347565642 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1802c9e0 con 0x7fba20140d00 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: dump: 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.099 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: _finish_auth 0 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: _check_auth_tickets 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 1 -- 172.21.15.64:0/347565642 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba1802cb80 con 0x7fba20140d00 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba28225640 5 monclient: authenticate success, global_id 16382 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba28225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 1 -- 172.21.15.64:0/347565642 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba180343f0 con 0x7fba20140d00 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba24fbf640 10 monclient: dump: 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.100 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba28225640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:28.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.099+0000 7fba28225640 4 set_mon_vals no callback set 2024-04-29T20:48:28.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 10 monclient: shutdown 2024-04-29T20:48:28.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 -- 172.21.15.64:0/347565642 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 msgr2=0x7fba20147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 --2- 172.21.15.64:0/347565642 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 0x7fba20147160 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fba18009e50 tx=0x7fba1802be40 comp rx=0 tx=0).stop 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 -- 172.21.15.64:0/347565642 shutdown_connections 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 --2- 172.21.15.64:0/347565642 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba20140d00 0x7fba20147160 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 --2- 172.21.15.64:0/347565642 >> v2:172.21.15.64:3300/0 conn(0x7fba2006cdb0 0x7fba201407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 -- 172.21.15.64:0/347565642 >> 172.21.15.64:0/347565642 conn(0x7fba200b9030 msgr2=0x7fba200b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 -- 172.21.15.64:0/347565642 shutdown_connections 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.100+0000 7fba28225640 1 -- 172.21.15.64:0/347565642 wait complete. 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 10 monclient: build_initial_monmap 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 10 monclient: monmap: 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 1 librados: starting msgr at 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 1 librados: starting objecter 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 1 Processor -- start 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 1 -- start start 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 1 librados: setting wanted keys 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 1 librados: calling monclient init 2024-04-29T20:48:28.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.101+0000 7fba28225640 10 monclient: init 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba2006cdb0 0x7fba20136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): picked mon.noname-c con 0x7fba2006cdb0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba200b3270 0x7fba20144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): picked mon.noname-b con 0x7fba200b3270 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba20140d00 0x7fba201dc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): picked mon.noname-a con 0x7fba20140d00 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fba200aace0 con 0x7fba20140d00 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fba200ab5d0 con 0x7fba200b3270 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fba200aae60 con 0x7fba2006cdb0 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba28225640 10 monclient(hunting): authenticate will time out at 3057.261830s 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.102+0000 7fba257c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba200b3270 0x7fba20144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba200b3270 0x7fba20144950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56650/0 (socket says 172.21.15.64:56650) 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 1 -- 172.21.15.64:0/242283150 learned_addr learned my addr 172.21.15.64:0/242283150 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): get_auth_request con 0x7fba200b3270 auth_method 0 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.103+0000 7fba257c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.104+0000 7fba25fc1640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba2006cdb0 0x7fba20136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.104+0000 7fba267c2640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba20140d00 0x7fba201dc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): get_auth_request con 0x7fba20140d00 auth_method 0 2024-04-29T20:48:28.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba25fc1640 10 monclient(hunting): get_auth_request con 0x7fba2006cdb0 auth_method 0 2024-04-29T20:48:28.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba25fc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba25fc1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba25fc1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.105+0000 7fba267c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba25fc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba25fc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba25fc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 10 monclient(hunting): handle_auth_done global_id 22177 payload 824 2024-04-29T20:48:28.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 1 -- 172.21.15.64:0/242283150 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba2006cdb0 msgr2=0x7fba20136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fba2006cdb0 0x7fba20136b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 1 -- 172.21.15.64:0/242283150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba20140d00 msgr2=0x7fba201dc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba20140d00 0x7fba201dc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 10 monclient: _finish_hunting 0 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 1 monclient: found mon.noname-b 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.106+0000 7fba257c0640 1 -- 172.21.15.64:0/242283150 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba20146520 con 0x7fba200b3270 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.108+0000 7fba257c0640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba200b3270 0x7fba20144950 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c440 tx=0x7fba1000c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.108+0000 7fba167fc640 10 client.?.objecter ms_handle_connect 0x7fba200b3270 2024-04-29T20:48:28.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.108+0000 7fba167fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10017070 con 0x7fba200b3270 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: dump: 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.112 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: _finish_auth 0 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: _check_auth_tickets 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba100035f0 con 0x7fba200b3270 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba100039e0 con 0x7fba200b3270 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 5 monclient: authenticate success, global_id 22177 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 10 monclient: _renew_subs 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba201415f0 con 0x7fba200b3270 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba17fff640 4 set_mon_vals no callback set 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba167fc640 10 monclient: dump: 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.114 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 10 client.22177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 10 monclient: _renew_subs 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba2013dbd0 con 0x7fba200b3270 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.112+0000 7fba28225640 1 librados: init done 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.113+0000 7fba28225640 10 client.22177.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.113+0000 7fba28225640 15 client.22177.objecter _session_command_op_assign -1 1 2024-04-29T20:48:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.113+0000 7fba28225640 20 client.22177.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:28.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.113+0000 7fba28225640 10 client.22177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:28.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.113+0000 7fba28225640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.113+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba2004fce0 con 0x7fba200b3270 2024-04-29T20:48:28.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.131+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba1001a070 con 0x7fba200b3270 2024-04-29T20:48:28.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.131+0000 7fba167fc640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fba0406e300 0x7fba040707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.131+0000 7fba25fc1640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fba0406e300 0x7fba040707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.131+0000 7fba25fc1640 10 monclient: get_auth_request con 0x7fba0406e300 auth_method 0 2024-04-29T20:48:28.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.132+0000 7fba25fc1640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fba0406e300 0x7fba040707c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009990 tx=0x7fba1c009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.132+0000 7fba167fc640 10 client.22177.objecter ms_handle_connect 0x7fba0406e300 2024-04-29T20:48:28.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7fba100573c0 con 0x7fba200b3270 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 10 client.22177.objecter ms_dispatch 0x7fba20000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 3 client.22177.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 3 client.22177.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 10 client.22177.objecter checking command 1 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7fba04076000 0x7fba040784a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 20 client.22177.objecter _get_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 15 client.22177.objecter _session_command_op_remove -1 1 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 20 client.22177.objecter _get_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 20 client.22177.objecter get_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 15 client.22177.objecter _session_command_op_assign 4 1 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 4 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 10 client.22177.objecter _send_command 1 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 --> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba10014ec0 con 0x7fba04076000 2024-04-29T20:48:28.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.134+0000 7fba167fc640 20 client.22177.objecter dump_active .. 0 homeless 2024-04-29T20:48:28.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.135+0000 7fba267c2640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7fba04076000 0x7fba040784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.136+0000 7fba267c2640 10 monclient: get_auth_request con 0x7fba04076000 auth_method 0 2024-04-29T20:48:28.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.138+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7fba100555c0 con 0x7fba200b3270 2024-04-29T20:48:28.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.138+0000 7fba167fc640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:28.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.138+0000 7fba267c2640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7fba04076000 0x7fba040784a0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.138+0000 7fba167fc640 10 client.22177.objecter ms_handle_connect 0x7fba04076000 2024-04-29T20:48:28.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.163+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== osd.4 v2:172.21.15.167:6826/3754201859 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba10014ec0 con 0x7fba04076000 2024-04-29T20:48:28.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.163+0000 7fba167fc640 10 client.22177.objecter ms_dispatch 0x7fba20000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:28.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.163+0000 7fba167fc640 10 client.22177.objecter _finish_command 1 = system:0 2024-04-29T20:48:28.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.163+0000 7fba167fc640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 4 2024-04-29T20:48:28.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.163+0000 7fba167fc640 15 client.22177.objecter _session_command_op_remove 4 1 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 10 client.22177.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 15 client.22177.objecter _session_command_op_assign -1 2 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 20 client.22177.objecter _get_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 20 client.22177.objecter _get_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 15 client.22177.objecter _session_command_op_remove -1 2 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 20 client.22177.objecter get_session s=0x7fba04075930 osd=4 3 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 15 client.22177.objecter _session_command_op_assign 4 2 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 4 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 10 client.22177.objecter _send_command 2 2024-04-29T20:48:28.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.176+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 --> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fba2013e4e0 con 0x7fba04076000 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba167fc640 1 -- 172.21.15.64:0/242283150 <== osd.4 v2:172.21.15.167:6826/3754201859 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fba2013e4e0 con 0x7fba04076000 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba167fc640 10 client.22177.objecter ms_dispatch 0x7fba20000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba167fc640 10 client.22177.objecter _finish_command 2 = system:0 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba167fc640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 4 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba167fc640 15 client.22177.objecter _session_command_op_remove 4 2 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:28.189 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 10 librados: watch_flush enter 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 10 librados: watch_flush exit 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 10 client.22177.objecter close_session for osd.4 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7fba04076000 msgr2=0x7fba040784a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6826/3754201859,v1:172.21.15.167:6827/3754201859] conn(0x7fba04076000 0x7fba040784a0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 20 client.22177.objecter put_session s=0x7fba04075930 osd=4 1 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 20 client.22177.objecter shutdown clearing up homeless session... 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 10 client.22177.objecter successfully canceled tick 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fba0406e300 msgr2=0x7fba040707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fba0406e300 0x7fba040707c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009990 tx=0x7fba1c009110 comp rx=0 tx=0).stop 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 10 monclient: shutdown 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba200b3270 msgr2=0x7fba20144950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 --2- 172.21.15.64:0/242283150 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba200b3270 0x7fba20144950 secure :-1 s=READY pgs=5635 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c440 tx=0x7fba1000c840 comp rx=0 tx=0).stop 2024-04-29T20:48:28.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba267c2640 1 -- 172.21.15.64:0/242283150 reap_dead start 2024-04-29T20:48:28.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 shutdown_connections 2024-04-29T20:48:28.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 >> 172.21.15.64:0/242283150 conn(0x7fba200b9030 msgr2=0x7fba20146f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:28.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 shutdown_connections 2024-04-29T20:48:28.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 -- 172.21.15.64:0/242283150 wait complete. 2024-04-29T20:48:28.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.189+0000 7fba28225640 1 librados: shutdown 2024-04-29T20:48:28.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-29T20:48:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.304+0000 7f18469b3640 10 monclient: get_monmap_and_config 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.304+0000 7f18469b3640 10 monclient: build_initial_monmap 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.304+0000 7f18469b3640 10 monclient: monmap: 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:28.304887+0000 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:28.304887+0000 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:28.304 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.304+0000 7f18469b3640 1 Processor -- start 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.304+0000 7f18469b3640 1 -- start start 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.304+0000 7f18469b3640 10 monclient: init 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f1840146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): picked mon.noname-b con 0x7f18400b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f18400aa460 0x7f18400b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): picked mon.noname-a con 0x7f18400aa460 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f18400b5c60 0x7f18400b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): picked mon.noname-c con 0x7f18400b5c60 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f1840146eb0 con 0x7f18400aa460 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f18400ac1f0 con 0x7f18400b5c60 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f18400ac4d0 con 0x7f18400b2230 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.305+0000 7f18469b3640 10 monclient(hunting): authenticate will time out at 3057.465095s 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f18400aa460 0x7f18400b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f1844f50640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f18400b5c60 0x7f18400b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183ffff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f1840146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f18400aa460 0x7f18400b2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36910/0 (socket says 172.21.15.64:36910) 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 1 -- 172.21.15.64:0/2874780038 learned_addr learned my addr 172.21.15.64:0/2874780038 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 10 monclient(hunting): get_auth_request con 0x7f18400aa460 auth_method 0 2024-04-29T20:48:28.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f1844f50640 10 monclient(hunting): get_auth_request con 0x7f18400b5c60 auth_method 0 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f1844f50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f1844f50640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f1844f50640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f18400b2230 auth_method 0 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.306+0000 7f183ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f1844f50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f1844f50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f1844f50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.307+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 10 monclient(hunting): handle_auth_done global_id 22180 payload 274 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 1 -- 172.21.15.64:0/2874780038 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 msgr2=0x7f1840146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 1 --2- 172.21.15.64:0/2874780038 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f1840146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 1 -- 172.21.15.64:0/2874780038 >> v2:172.21.15.64:3300/0 conn(0x7f18400aa460 msgr2=0x7f18400b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 1 --2- 172.21.15.64:0/2874780038 >> v2:172.21.15.64:3300/0 conn(0x7f18400aa460 0x7f18400b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 10 monclient: _finish_hunting 0 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 1 monclient: found mon.noname-c 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.308+0000 7f1844f50640 1 -- 172.21.15.64:0/2874780038 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18400b3440 con 0x7f18400b5c60 2024-04-29T20:48:28.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.310+0000 7f1844f50640 1 --2- 172.21.15.64:0/2874780038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18400b37b0 secure :-1 s=READY pgs=5636 cs=0 l=1 rev1=1 crypto rx=0x7f184006daa0 tx=0x7f183802bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e27eec556ba38a08 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 1 -- 172.21.15.64:0/2874780038 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838039070 con 0x7f18400b5c60 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: dump: 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.314 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: _finish_auth 0 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 1 -- 172.21.15.64:0/2874780038 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f183802cb40 con 0x7f18400b5c60 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 1 -- 172.21.15.64:0/2874780038 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838034440 con 0x7f18400b5c60 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 5 monclient: authenticate success, global_id 22180 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 4 set_mon_vals no callback set 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f183effd640 10 monclient: dump: 2024-04-29T20:48:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.316 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 10 monclient: shutdown 2024-04-29T20:48:28.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:28.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 -- 172.21.15.64:0/2874780038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 msgr2=0x7f18400b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 --2- 172.21.15.64:0/2874780038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18400b37b0 secure :-1 s=READY pgs=5636 cs=0 l=1 rev1=1 crypto rx=0x7f184006daa0 tx=0x7f183802bd90 comp rx=0 tx=0).stop 2024-04-29T20:48:28.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 -- 172.21.15.64:0/2874780038 shutdown_connections 2024-04-29T20:48:28.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 --2- 172.21.15.64:0/2874780038 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18400b37b0 unknown :-1 s=CLOSED pgs=5636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 --2- 172.21.15.64:0/2874780038 >> v2:172.21.15.64:3300/0 conn(0x7f18400aa460 0x7f18400b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 --2- 172.21.15.64:0/2874780038 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f1840146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.314+0000 7f18469b3640 1 -- 172.21.15.64:0/2874780038 >> 172.21.15.64:0/2874780038 conn(0x7f184006c960 msgr2=0x7f184006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.315+0000 7f18469b3640 1 -- 172.21.15.64:0/2874780038 shutdown_connections 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.315+0000 7f18469b3640 1 -- 172.21.15.64:0/2874780038 wait complete. 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.315+0000 7f18469b3640 10 monclient: build_initial_monmap 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.315+0000 7f18469b3640 10 monclient: monmap: 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.320 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 librados: starting msgr at 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 librados: starting objecter 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 Processor -- start 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 -- start start 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 librados: setting wanted keys 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 librados: calling monclient init 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient: init 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400aa460 0x7f18400b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): picked mon.noname-a con 0x7f18400aa460 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f18400b9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): picked mon.noname-c con 0x7f18400b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18401d90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): picked mon.noname-b con 0x7f18400b5c60 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f18400b6250 con 0x7f18400aa460 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f18400b60d0 con 0x7f18400b5c60 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f18400b2610 con 0x7f18400b2230 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.316+0000 7f18469b3640 10 monclient(hunting): authenticate will time out at 3057.476540s 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400aa460 0x7f18400b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f1844f50640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18401d90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183f7fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f18400b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400aa460 0x7f18400b12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36914/0 (socket says 172.21.15.64:36914) 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 1 -- 172.21.15.64:0/1162178042 learned_addr learned my addr 172.21.15.64:0/1162178042 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f18400aa460 auth_method 0 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.317+0000 7f183ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): get_auth_request con 0x7f18400b5c60 auth_method 0 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): get_auth_request con 0x7f18400b2230 auth_method 0 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f1844f50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.318+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 10 monclient(hunting): handle_auth_done global_id 22183 payload 824 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 1 -- 172.21.15.64:0/1162178042 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 msgr2=0x7f18400b9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f18400b9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 1 -- 172.21.15.64:0/1162178042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400aa460 msgr2=0x7f18400b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400aa460 0x7f18400b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 10 monclient: _finish_hunting 0 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 1 monclient: found mon.noname-b 2024-04-29T20:48:28.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:28.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.321+0000 7f1844f50640 1 -- 172.21.15.64:0/1162178042 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18400adea0 con 0x7f18400b5c60 2024-04-29T20:48:28.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.323+0000 7f1844f50640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18401d90d0 secure :-1 s=READY pgs=5637 cs=0 l=1 rev1=1 crypto rx=0x7f1838033000 tx=0x7f1838004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.323+0000 7f183cff9640 10 client.?.objecter ms_handle_connect 0x7f18400b5c60 2024-04-29T20:48:28.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.323+0000 7f183cff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.326+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838039050 con 0x7f18400b5c60 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.326+0000 7f183cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.326+0000 7f183cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.326+0000 7f183cff9640 10 monclient: dump: 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 10 monclient: _finish_auth 0 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 10 monclient: _check_auth_tickets 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1838045070 con 0x7f18400b5c60 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 5 monclient: authenticate success, global_id 22183 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 10 monclient: _renew_subs 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1840143320 con 0x7f18400b5c60 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838009450 con 0x7f18400b5c60 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183cff9640 10 monclient: dump: 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f183e7fc640 4 set_mon_vals no callback set 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 10 client.22183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 10 monclient: _renew_subs 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18400aca20 con 0x7f18400b5c60 2024-04-29T20:48:28.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.327+0000 7f18469b3640 1 librados: init done 2024-04-29T20:48:28.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.328+0000 7f18469b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:28.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.328+0000 7f18469b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:28.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.328+0000 7f18469b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:28.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.328+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1840050530 con 0x7f18400b5c60 2024-04-29T20:48:28.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.346+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f183803e850 con 0x7f18400b5c60 2024-04-29T20:48:28.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.346+0000 7f183cff9640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f182006e280 0x7f1820070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.347+0000 7f183ffff640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f182006e280 0x7f1820070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.347+0000 7f183ffff640 10 monclient: get_auth_request con 0x7f182006e280 auth_method 0 2024-04-29T20:48:28.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.347+0000 7f183cff9640 10 client.22183.objecter ms_handle_connect 0x7f182006e280 2024-04-29T20:48:28.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.347+0000 7f183ffff640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f182006e280 0x7f1820070740 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1830009c00 tx=0x7f1830009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.581+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f183807af60 con 0x7f18400b5c60 2024-04-29T20:48:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.581+0000 7f183cff9640 10 client.22183.objecter ms_dispatch 0x7f1840000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.581+0000 7f183cff9640 3 client.22183.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.581+0000 7f183cff9640 3 client.22183.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.581+0000 7f183cff9640 20 client.22183.objecter dump_active .. 0 homeless 2024-04-29T20:48:28.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.591+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1840050530 con 0x7f18400b5c60 2024-04-29T20:48:28.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.591+0000 7f183cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:28.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.591+0000 7f183cff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.794+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f184004fce0 con 0x7f182006e280 2024-04-29T20:48:28.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.794+0000 7f183cff9640 1 -- 172.21.15.64:0/1162178042 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f184004fce0 con 0x7f182006e280 2024-04-29T20:48:28.794 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 4 to deep-scrub 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 10 librados: watch_flush enter 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 10 librados: watch_flush exit 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 20 client.22183.objecter shutdown clearing up homeless session... 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 10 client.22183.objecter successfully canceled tick 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f182006e280 msgr2=0x7f1820070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f182006e280 0x7f1820070740 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1830009c00 tx=0x7f1830009190 comp rx=0 tx=0).stop 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 10 monclient: shutdown 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 msgr2=0x7f18401d90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18401d90d0 secure :-1 s=READY pgs=5637 cs=0 l=1 rev1=1 crypto rx=0x7f1838033000 tx=0x7f1838004a40 comp rx=0 tx=0).stop 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 shutdown_connections 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f182006e280 0x7f1820070740 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f18400b2230 0x7f18400b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18400b5c60 0x7f18401d90d0 unknown :-1 s=CLOSED pgs=5637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 --2- 172.21.15.64:0/1162178042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400aa460 0x7f18400b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 >> 172.21.15.64:0/1162178042 conn(0x7f184006c960 msgr2=0x7f18400b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:28.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 shutdown_connections 2024-04-29T20:48:28.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.796+0000 7f18469b3640 1 -- 172.21.15.64:0/1162178042 wait complete. 2024-04-29T20:48:28.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.797+0000 7f18469b3640 1 librados: shutdown 2024-04-29T20:48:28.807 INFO:tasks.ceph:Scrubbing osd.5 2024-04-29T20:48:28.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.921+0000 7f15d2e69640 10 monclient: get_monmap_and_config 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.921+0000 7f15d2e69640 10 monclient: build_initial_monmap 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.921+0000 7f15d2e69640 10 monclient: monmap: 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:28.922383+0000 2024-04-29T20:48:28.921 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:28.922383+0000 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:28.922 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 Processor -- start 2024-04-29T20:48:28.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 -- start start 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient: init 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc0b5b90 0x7f15cc146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): picked mon.noname-b con 0x7f15cc0b5b90 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15cc0b5f70 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:28.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f15cc0aa460 0x7f15cc0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): picked mon.noname-c con 0x7f15cc0aa460 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15cc0aa910 con 0x7f15cc0b5f70 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f15cc0aabc0 con 0x7f15cc0aa460 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f15cc0b5450 con 0x7f15cc0b5b90 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.922+0000 7f15d2e69640 10 monclient(hunting): authenticate will time out at 3058.082554s 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.923+0000 7f15d0c05640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc0b5b90 0x7f15cc146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.923+0000 7f15cbfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f15cc0b5f70 0x7f15cc0b2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60516/0 (socket says 172.21.15.64:60516) 2024-04-29T20:48:28.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.923+0000 7f15cbfff640 1 -- 172.21.15.64:0/2804592446 learned_addr learned my addr 172.21.15.64:0/2804592446 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.923+0000 7f15d1406640 1 --2- 172.21.15.64:0/2804592446 >> v2:172.21.15.167:3300/0 conn(0x7f15cc0aa460 0x7f15cc0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.923+0000 7f15d0c05640 10 monclient(hunting): get_auth_request con 0x7f15cc0b5b90 auth_method 0 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d0c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d0c05640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d0c05640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d1406640 10 monclient(hunting): get_auth_request con 0x7f15cc0aa460 auth_method 0 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d1406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d1406640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d1406640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d0c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d0c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.924+0000 7f15d0c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.925+0000 7f15d1406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.925+0000 7f15d1406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.925+0000 7f15d1406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 10 monclient(hunting): handle_auth_done global_id 16394 payload 274 2024-04-29T20:48:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 1 -- 172.21.15.64:0/2804592446 >> v2:172.21.15.167:3300/0 conn(0x7f15cc0aa460 msgr2=0x7f15cc0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 1 --2- 172.21.15.64:0/2804592446 >> v2:172.21.15.167:3300/0 conn(0x7f15cc0aa460 0x7f15cc0b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 1 -- 172.21.15.64:0/2804592446 >> v1:172.21.15.64:6789/0 conn(0x7f15cc0b5f70 legacy=0x7f15cc0b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 10 monclient: _finish_hunting 0 2024-04-29T20:48:28.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 1 monclient: found mon.noname-b 2024-04-29T20:48:28.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:28.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:28.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.926+0000 7f15d0c05640 1 -- 172.21.15.64:0/2804592446 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc0b5600 con 0x7f15cc0b5b90 2024-04-29T20:48:28.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.928+0000 7f15d0c05640 1 --2- 172.21.15.64:0/2804592446 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc0b5b90 0x7f15cc146900 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f15c00097a0 tx=0x7f15c002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29248cc49899ebcd server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 1 -- 172.21.15.64:0/2804592446 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0035020 con 0x7f15cc0b5b90 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: dump: 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.932 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: _finish_auth 0 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 5 monclient: authenticate success, global_id 16394 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 1 -- 172.21.15.64:0/2804592446 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c0031830 con 0x7f15cc0b5b90 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 4 set_mon_vals no callback set 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 1 -- 172.21.15.64:0/2804592446 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0031c00 con 0x7f15cc0b5b90 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15cb7fe640 10 monclient: dump: 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.934 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 10 monclient: shutdown 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 1 -- 172.21.15.64:0/2804592446 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc0b5b90 msgr2=0x7f15cc146900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.932+0000 7f15d2e69640 1 --2- 172.21.15.64:0/2804592446 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc0b5b90 0x7f15cc146900 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f15c00097a0 tx=0x7f15c002b8d0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 1 -- 172.21.15.64:0/2804592446 shutdown_connections 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 1 --2- 172.21.15.64:0/2804592446 >> v2:172.21.15.167:3300/0 conn(0x7f15cc0aa460 0x7f15cc0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 1 --2- 172.21.15.64:0/2804592446 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc0b5b90 0x7f15cc146900 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 1 -- 172.21.15.64:0/2804592446 >> 172.21.15.64:0/2804592446 conn(0x7f15cc06c9d0 msgr2=0x7f15cc06cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 1 -- 172.21.15.64:0/2804592446 shutdown_connections 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 1 -- 172.21.15.64:0/2804592446 wait complete. 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 10 monclient: build_initial_monmap 2024-04-29T20:48:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.933+0000 7f15d2e69640 10 monclient: monmap: 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 librados: starting msgr at 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 librados: starting objecter 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 Processor -- start 2024-04-29T20:48:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 -- start start 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 librados: setting wanted keys 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 librados: calling monclient init 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient: init 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15cc0aa460 0x7f15cc13f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): picked mon.noname-b con 0x7f15cc0aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15cc0b5b90 0x7f15cc141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): picked mon.noname-a con 0x7f15cc0b5b90 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc142990 0x7f15cc143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): picked mon.noname-c con 0x7f15cc142990 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc0b2570 con 0x7f15cc0b5b90 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc14dc90 con 0x7f15cc0aa460 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f15cc1432a0 con 0x7f15cc142990 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.934+0000 7f15d2e69640 10 monclient(hunting): authenticate will time out at 3058.094554s 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.935+0000 7f15cbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15cc0b5b90 0x7f15cc141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.935+0000 7f15d1406640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc142990 0x7f15cc143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.935+0000 7f15d0c05640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15cc0aa460 0x7f15cc13f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.935+0000 7f15d1406640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc142990 0x7f15cc143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45754/0 (socket says 172.21.15.64:45754) 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.935+0000 7f15d1406640 1 -- 172.21.15.64:0/2688551591 learned_addr learned my addr 172.21.15.64:0/2688551591 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): get_auth_request con 0x7f15cc142990 auth_method 0 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): get_auth_request con 0x7f15cc0aa460 auth_method 0 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): get_auth_request con 0x7f15cc0b5b90 auth_method 0 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d0c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15d1406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.936+0000 7f15cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.940+0000 7f15d1406640 10 monclient(hunting): handle_auth_done global_id 16397 payload 824 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 1 -- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15cc0aa460 msgr2=0x7f15cc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15cc0aa460 0x7f15cc13f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 1 -- 172.21.15.64:0/2688551591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15cc0b5b90 msgr2=0x7f15cc141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15cc0b5b90 0x7f15cc141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 10 monclient: _finish_hunting 0 2024-04-29T20:48:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 1 monclient: found mon.noname-c 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d1406640 1 -- 172.21.15.64:0/2688551591 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc0ad110 con 0x7f15cc142990 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.941+0000 7f15d0c05640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15cc0aa460 0x7f15cc13f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.943+0000 7f15d1406640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc142990 0x7f15cc143730 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009dd0 tx=0x7f15c400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.943+0000 7f15c97fa640 10 client.?.objecter ms_handle_connect 0x7f15cc142990 2024-04-29T20:48:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.943+0000 7f15c97fa640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:28.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4018020 con 0x7f15cc142990 2024-04-29T20:48:28.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: dump: 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: _finish_auth 0 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: _check_auth_tickets 2024-04-29T20:48:28.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c4014480 con 0x7f15cc142990 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 5 monclient: authenticate success, global_id 16397 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 10 monclient: _renew_subs 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4014870 con 0x7f15cc142990 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc0ad380 con 0x7f15cc142990 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15c97fa640 10 monclient: dump: 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:28.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15caffd640 4 set_mon_vals no callback set 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 10 client.16397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 10 monclient: _renew_subs 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc143c70 con 0x7f15cc142990 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.947+0000 7f15d2e69640 1 librados: init done 2024-04-29T20:48:28.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.948+0000 7f15d2e69640 10 client.16397.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:28.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.948+0000 7f15d2e69640 15 client.16397.objecter _session_command_op_assign -1 1 2024-04-29T20:48:28.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.948+0000 7f15d2e69640 20 client.16397.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:28.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.948+0000 7f15d2e69640 10 client.16397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:28.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.948+0000 7f15d2e69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:28.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.948+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15cc04fce0 con 0x7f15cc142990 2024-04-29T20:48:28.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.970+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f15c4005d10 con 0x7f15cc142990 2024-04-29T20:48:28.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.970+0000 7f15c97fa640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f15b006e2d0 0x7f15b0070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.971+0000 7f15d0c05640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f15b006e2d0 0x7f15b0070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.971+0000 7f15d0c05640 10 monclient: get_auth_request con 0x7f15b006e2d0 auth_method 0 2024-04-29T20:48:28.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.971+0000 7f15c97fa640 10 client.16397.objecter ms_handle_connect 0x7f15b006e2d0 2024-04-29T20:48:28.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.971+0000 7f15d0c05640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f15b006e2d0 0x7f15b0070790 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f15c0036000 tx=0x7f15c0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:28.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f15c4058fb0 con 0x7f15cc142990 2024-04-29T20:48:28.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 10 client.16397.objecter ms_dispatch 0x7f15cc000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 3 client.16397.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 3 client.16397.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 10 client.16397.objecter checking command 1 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7f15b0075fd0 0x7f15b0078470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 20 client.16397.objecter _get_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 15 client.16397.objecter _session_command_op_remove -1 1 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 20 client.16397.objecter _get_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 20 client.16397.objecter get_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 15 client.16397.objecter _session_command_op_assign 5 1 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 4 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 10 client.16397.objecter _send_command 1 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 --> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15c400df90 con 0x7f15b0075fd0 2024-04-29T20:48:28.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 20 client.16397.objecter dump_active .. 0 homeless 2024-04-29T20:48:28.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7f15c40571b0 con 0x7f15cc142990 2024-04-29T20:48:28.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.976+0000 7f15c97fa640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:28.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.979+0000 7f15cbfff640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7f15b0075fd0 0x7f15b0078470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:28.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.980+0000 7f15cbfff640 10 monclient: get_auth_request con 0x7f15b0075fd0 auth_method 0 2024-04-29T20:48:28.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.985+0000 7f15c97fa640 10 client.16397.objecter ms_handle_connect 0x7f15b0075fd0 2024-04-29T20:48:28.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:28.985+0000 7f15cbfff640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7f15b0075fd0 0x7f15b0078470 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:29.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.010+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== osd.5 v2:172.21.15.167:6802/3447508053 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15c400df90 con 0x7f15b0075fd0 2024-04-29T20:48:29.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.010+0000 7f15c97fa640 10 client.16397.objecter ms_dispatch 0x7f15cc000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:29.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.010+0000 7f15c97fa640 10 client.16397.objecter _finish_command 1 = system:0 2024-04-29T20:48:29.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.010+0000 7f15c97fa640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 4 2024-04-29T20:48:29.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.010+0000 7f15c97fa640 15 client.16397.objecter _session_command_op_remove 5 1 2024-04-29T20:48:29.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 10 client.16397.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:29.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 15 client.16397.objecter _session_command_op_assign -1 2 2024-04-29T20:48:29.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 20 client.16397.objecter _get_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:29.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:29.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 20 client.16397.objecter _get_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 15 client.16397.objecter _session_command_op_remove -1 2 2024-04-29T20:48:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 20 client.16397.objecter get_session s=0x7f15b0075900 osd=5 3 2024-04-29T20:48:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 15 client.16397.objecter _session_command_op_assign 5 2 2024-04-29T20:48:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 4 2024-04-29T20:48:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 10 client.16397.objecter _send_command 2 2024-04-29T20:48:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.025+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 --> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f15cc143420 con 0x7f15b0075fd0 2024-04-29T20:48:29.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15c97fa640 1 -- 172.21.15.64:0/2688551591 <== osd.5 v2:172.21.15.167:6802/3447508053 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f15cc143420 con 0x7f15b0075fd0 2024-04-29T20:48:29.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15c97fa640 10 client.16397.objecter ms_dispatch 0x7f15cc000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:29.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15c97fa640 10 client.16397.objecter _finish_command 2 = system:0 2024-04-29T20:48:29.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15c97fa640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 4 2024-04-29T20:48:29.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15c97fa640 15 client.16397.objecter _session_command_op_remove 5 2 2024-04-29T20:48:29.045 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:29.046 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:29.046 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 10 librados: watch_flush enter 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 10 librados: watch_flush exit 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 10 client.16397.objecter close_session for osd.5 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7f15b0075fd0 msgr2=0x7f15b0078470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6802/3447508053,v1:172.21.15.167:6803/3447508053] conn(0x7f15b0075fd0 0x7f15b0078470 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 20 client.16397.objecter put_session s=0x7f15b0075900 osd=5 1 2024-04-29T20:48:29.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 20 client.16397.objecter shutdown clearing up homeless session... 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 10 client.16397.objecter successfully canceled tick 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f15b006e2d0 msgr2=0x7f15b0070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f15b006e2d0 0x7f15b0070790 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f15c0036000 tx=0x7f15c0008030 comp rx=0 tx=0).stop 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 10 monclient: shutdown 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc142990 msgr2=0x7f15cc143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d2e69640 1 --2- 172.21.15.64:0/2688551591 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f15cc142990 0x7f15cc143730 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009dd0 tx=0x7f15c400c490 comp rx=0 tx=0).stop 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.045+0000 7f15d1406640 1 -- 172.21.15.64:0/2688551591 reap_dead start 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.046+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 shutdown_connections 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.046+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 >> 172.21.15.64:0/2688551591 conn(0x7f15cc06c9d0 msgr2=0x7f15cc0b6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.046+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 shutdown_connections 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.046+0000 7f15d2e69640 1 -- 172.21.15.64:0/2688551591 wait complete. 2024-04-29T20:48:29.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.046+0000 7f15d2e69640 1 librados: shutdown 2024-04-29T20:48:29.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.167+0000 7f17b66a8640 10 monclient: get_monmap_and_config 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.167+0000 7f17b66a8640 10 monclient: build_initial_monmap 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.167+0000 7f17b66a8640 10 monclient: monmap: 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:29.168241+0000 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:29.168241+0000 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.167 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:29.168 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:29.168 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:29.168 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:29.168 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:29.168 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 Processor -- start 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 -- start start 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient: init 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f17b00b22a0 0x7f17b0146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): picked mon.noname-c con 0x7f17b00b22a0 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17b00b2c20 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:29.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f17b00aa6a0 0x7f17b014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): picked mon.noname-a con 0x7f17b00aa6a0 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b00ac1f0 con 0x7f17b00b2c20 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f17b00b33d0 con 0x7f17b00aa6a0 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f17b00ac4d0 con 0x7f17b00b22a0 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:29.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.168+0000 7f17b66a8640 10 monclient(hunting): authenticate will time out at 3058.328419s 2024-04-29T20:48:29.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.192+0000 7f17affff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f17b00b22a0 0x7f17b0146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f17b00b22a0 0x7f17b0146a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56676/0 (socket says 172.21.15.64:56676) 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 1 -- 172.21.15.64:0/2089491833 learned_addr learned my addr 172.21.15.64:0/2089491833 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): get_auth_request con 0x7f17b00b22a0 auth_method 0 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.193+0000 7f17affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.195+0000 7f17b4c45640 1 --2- 172.21.15.64:0/2089491833 >> v2:172.21.15.64:3300/0 conn(0x7f17b00aa6a0 0x7f17b014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 10 monclient(hunting): handle_auth_done global_id 22192 payload 274 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 1 -- 172.21.15.64:0/2089491833 >> v2:172.21.15.64:3300/0 conn(0x7f17b00aa6a0 msgr2=0x7f17b014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 1 --2- 172.21.15.64:0/2089491833 >> v2:172.21.15.64:3300/0 conn(0x7f17b00aa6a0 0x7f17b014ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 1 -- 172.21.15.64:0/2089491833 >> v1:172.21.15.64:6789/0 conn(0x7f17b00b2c20 legacy=0x7f17b00b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 10 monclient: _finish_hunting 0 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 1 monclient: found mon.noname-c 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:29.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.196+0000 7f17affff640 1 -- 172.21.15.64:0/2089491833 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b00b3040 con 0x7f17b00b22a0 2024-04-29T20:48:29.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.198+0000 7f17affff640 1 --2- 172.21.15.64:0/2089491833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0146a40 secure :-1 s=READY pgs=5638 cs=0 l=1 rev1=1 crypto rx=0x7f17b006d6e0 tx=0x7f17a002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ff77bdd6219ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 1 -- 172.21.15.64:0/2089491833 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0035020 con 0x7f17b00b22a0 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: dump: 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:29.203 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: _finish_auth 0 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 1 -- 172.21.15.64:0/2089491833 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a0031870 con 0x7f17b00b22a0 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17b66a8640 5 monclient: authenticate success, global_id 22192 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17b66a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17b66a8640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 1 -- 172.21.15.64:0/2089491833 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0031c40 con 0x7f17b00b22a0 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17b66a8640 4 set_mon_vals no callback set 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.203+0000 7f17aeffd640 10 monclient: dump: 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:29.205 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 10 monclient: shutdown 2024-04-29T20:48:29.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:29.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 -- 172.21.15.64:0/2089491833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 msgr2=0x7f17b0146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 --2- 172.21.15.64:0/2089491833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0146a40 secure :-1 s=READY pgs=5638 cs=0 l=1 rev1=1 crypto rx=0x7f17b006d6e0 tx=0x7f17a002b910 comp rx=0 tx=0).stop 2024-04-29T20:48:29.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 -- 172.21.15.64:0/2089491833 shutdown_connections 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 --2- 172.21.15.64:0/2089491833 >> v2:172.21.15.64:3300/0 conn(0x7f17b00aa6a0 0x7f17b014ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 --2- 172.21.15.64:0/2089491833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0146a40 unknown :-1 s=CLOSED pgs=5638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 -- 172.21.15.64:0/2089491833 >> 172.21.15.64:0/2089491833 conn(0x7f17b006c940 msgr2=0x7f17b006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 -- 172.21.15.64:0/2089491833 shutdown_connections 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.204+0000 7f17b66a8640 1 -- 172.21.15.64:0/2089491833 wait complete. 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.205+0000 7f17b66a8640 10 monclient: build_initial_monmap 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.205+0000 7f17b66a8640 10 monclient: monmap: 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:29.209 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.205+0000 7f17b66a8640 1 librados: starting msgr at 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.205+0000 7f17b66a8640 1 librados: starting objecter 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.205+0000 7f17b66a8640 1 Processor -- start 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 -- start start 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 librados: setting wanted keys 2024-04-29T20:48:29.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 librados: calling monclient init 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient: init 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f17b00aa6a0 0x7f17b00b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): picked mon.noname-c con 0x7f17b00aa6a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): picked mon.noname-b con 0x7f17b00b22a0 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17b00b2c20 0x7f17b01dc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): picked mon.noname-a con 0x7f17b00b2c20 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f17b00aab30 con 0x7f17b00b2c20 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f17b00b2890 con 0x7f17b00b22a0 2024-04-29T20:48:29.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f17b00aacb0 con 0x7f17b00aa6a0 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.206+0000 7f17b66a8640 10 monclient(hunting): authenticate will time out at 3058.366461s 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0143210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56680/0 (socket says 172.21.15.64:56680) 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 1 -- 172.21.15.64:0/304518208 learned_addr learned my addr 172.21.15.64:0/304518208 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 10 monclient(hunting): get_auth_request con 0x7f17b00b22a0 auth_method 0 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.207+0000 7f17af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17affff640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f17b00aa6a0 0x7f17b00b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17affff640 10 monclient(hunting): get_auth_request con 0x7f17b00aa6a0 auth_method 0 2024-04-29T20:48:29.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17affff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.208+0000 7f17affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.209+0000 7f17affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.209+0000 7f17affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.209+0000 7f17affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 10 monclient(hunting): handle_auth_done global_id 22195 payload 824 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 1 -- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f17b00aa6a0 msgr2=0x7f17b00b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f17b00aa6a0 0x7f17b00b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 1 -- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17b00b2c20 msgr2=0x7f17b01dc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17b00b2c20 0x7f17b01dc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 10 monclient: _finish_hunting 0 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 1 monclient: found mon.noname-b 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:29.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.212+0000 7f17af7fe640 1 -- 172.21.15.64:0/304518208 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b0142870 con 0x7f17b00b22a0 2024-04-29T20:48:29.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.218+0000 7f17af7fe640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0143210 secure :-1 s=READY pgs=5639 cs=0 l=1 rev1=1 crypto rx=0x7f17a400b410 tx=0x7f17a400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:29.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.218+0000 7f17acff9640 10 client.?.objecter ms_handle_connect 0x7f17b00b22a0 2024-04-29T20:48:29.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.218+0000 7f17acff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4015020 con 0x7f17b00b22a0 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: dump: 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:29.247 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: _finish_auth 0 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: _check_auth_tickets 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a4011830 con 0x7f17b00b22a0 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4011c20 con 0x7f17b00b22a0 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17acff9640 10 monclient: dump: 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:29.248 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17b66a8640 5 monclient: authenticate success, global_id 22195 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17b66a8640 10 monclient: _renew_subs 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17b66a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b0143750 con 0x7f17b00b22a0 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.246+0000 7f17ae7fc640 4 set_mon_vals no callback set 2024-04-29T20:48:29.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 10 client.22195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:29.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 10 monclient: _renew_subs 2024-04-29T20:48:29.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:29.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b01dcb10 con 0x7f17b00b22a0 2024-04-29T20:48:29.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 1 librados: init done 2024-04-29T20:48:29.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:29.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:29.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:29.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.247+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b0143bc0 con 0x7f17b00b22a0 2024-04-29T20:48:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.266+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f17a4018020 con 0x7f17b00b22a0 2024-04-29T20:48:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.266+0000 7f17acff9640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f179c06e280 0x7f179c070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.266+0000 7f17affff640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f179c06e280 0x7f179c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.267+0000 7f17affff640 10 monclient: get_auth_request con 0x7f179c06e280 auth_method 0 2024-04-29T20:48:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.267+0000 7f17affff640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f179c06e280 0x7f179c070740 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f17a0036000 tx=0x7f17a003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.267+0000 7f17acff9640 10 client.22195.objecter ms_handle_connect 0x7f179c06e280 2024-04-29T20:48:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.268+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f17a4057970 con 0x7f17b00b22a0 2024-04-29T20:48:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.268+0000 7f17acff9640 10 client.22195.objecter ms_dispatch 0x7f17b0000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.268+0000 7f17acff9640 3 client.22195.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.268+0000 7f17acff9640 3 client.22195.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.268+0000 7f17acff9640 20 client.22195.objecter dump_active .. 0 homeless 2024-04-29T20:48:29.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.511+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17a401f030 con 0x7f17b00b22a0 2024-04-29T20:48:29.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.511+0000 7f17acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:29.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.511+0000 7f17acff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:29.631 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:48.138 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.644 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:48.151 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.681 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:48.159 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.687 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:48.194 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.689 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:48.197 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.698 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:48.206 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.711 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:48.189 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.725 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:48.204 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.734+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f17b004fce0 con 0x7f179c06e280 2024-04-29T20:48:29.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.735+0000 7f17acff9640 1 -- 172.21.15.64:0/304518208 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f17b004fce0 con 0x7f179c06e280 2024-04-29T20:48:29.735 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 5 to deep-scrub 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 10 librados: watch_flush enter 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 10 librados: watch_flush exit 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 20 client.22195.objecter shutdown clearing up homeless session... 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 10 client.22195.objecter successfully canceled tick 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f179c06e280 msgr2=0x7f179c070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f179c06e280 0x7f179c070740 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f17a0036000 tx=0x7f17a003c040 comp rx=0 tx=0).stop 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 10 monclient: shutdown 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 msgr2=0x7f17b0143210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0143210 secure :-1 s=READY pgs=5639 cs=0 l=1 rev1=1 crypto rx=0x7f17a400b410 tx=0x7f17a400b8d0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 shutdown_connections 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f179c06e280 0x7f179c070740 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17b00b2c20 0x7f17b01dc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17b00b22a0 0x7f17b0143210 unknown :-1 s=CLOSED pgs=5639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 --2- 172.21.15.64:0/304518208 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f17b00aa6a0 0x7f17b00b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.737+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 >> 172.21.15.64:0/304518208 conn(0x7f17b006c940 msgr2=0x7f17b00b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.738+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 shutdown_connections 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.738+0000 7f17b66a8640 1 -- 172.21.15.64:0/304518208 wait complete. 2024-04-29T20:48:29.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.738+0000 7f17b66a8640 1 librados: shutdown 2024-04-29T20:48:29.744 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:48.252 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:29.752 INFO:tasks.ceph:Scrubbing osd.6 2024-04-29T20:48:29.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:29.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.878+0000 7fa4a01ce640 10 monclient: get_monmap_and_config 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.878+0000 7fa4a01ce640 10 monclient: build_initial_monmap 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.878+0000 7fa4a01ce640 10 monclient: monmap: 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:29.879641+0000 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:29.879641+0000 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:29.879 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.879+0000 7fa4a01ce640 1 Processor -- start 2024-04-29T20:48:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.879+0000 7fa4a01ce640 1 -- start start 2024-04-29T20:48:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.879+0000 7fa4a01ce640 10 monclient: init 2024-04-29T20:48:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-29T20:48:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa49806cdb0 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7fa4980b3270 0x7fa498146a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): picked mon.noname-c con 0x7fa4980b3270 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498146f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): picked mon.noname-b con 0x7fa498140ed0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4980abff0 con 0x7fa49806cdb0 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7fa49806d190 con 0x7fa4980b3270 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa4980ab5d0 con 0x7fa498140ed0 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:29.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.880+0000 7fa4a01ce640 10 monclient(hunting): authenticate will time out at 3059.040187s 2024-04-29T20:48:29.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.886+0000 7fa49e76b640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.887+0000 7fa49e76b640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498146f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45778/0 (socket says 172.21.15.64:45778) 2024-04-29T20:48:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.887+0000 7fa49e76b640 1 -- 172.21.15.64:0/927615532 learned_addr learned my addr 172.21.15.64:0/927615532 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.887+0000 7fa49e76b640 10 monclient(hunting): get_auth_request con 0x7fa498140ed0 auth_method 0 2024-04-29T20:48:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.887+0000 7fa49e76b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.887+0000 7fa49e76b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.887+0000 7fa49e76b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.890+0000 7fa49e76b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.890+0000 7fa49e76b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.890+0000 7fa49e76b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 10 monclient(hunting): handle_auth_done global_id 16403 payload 274 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 1 -- 172.21.15.64:0/927615532 >> v2:172.21.15.167:3300/0 conn(0x7fa4980b3270 msgr2=0x7fa498146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 1 --2- 172.21.15.64:0/927615532 >> v2:172.21.15.167:3300/0 conn(0x7fa4980b3270 0x7fa498146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 1 -- 172.21.15.64:0/927615532 >> v1:172.21.15.64:6789/0 conn(0x7fa49806cdb0 legacy=0x7fa4981407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 10 monclient: _finish_hunting 0 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 1 monclient: found mon.noname-b 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:29.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.893+0000 7fa49e76b640 1 -- 172.21.15.64:0/927615532 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4980ac2d0 con 0x7fa498140ed0 2024-04-29T20:48:29.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.940+0000 7fa49e76b640 1 --2- 172.21.15.64:0/927615532 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498146f90 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa490009e50 tx=0x7fa49002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3899e9661eeda87d server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 1 -- 172.21.15.64:0/927615532 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49002c9e0 con 0x7fa498140ed0 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: dump: 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:29.949 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: _finish_auth 0 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: _check_auth_tickets 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 1 -- 172.21.15.64:0/927615532 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49002cb80 con 0x7fa498140ed0 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa4a01ce640 5 monclient: authenticate success, global_id 16403 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa4a01ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 1 -- 172.21.15.64:0/927615532 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4900343f0 con 0x7fa498140ed0 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa4a01ce640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa4a01ce640 4 set_mon_vals no callback set 2024-04-29T20:48:29.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.949+0000 7fa49cf68640 10 monclient: dump: 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:29.951 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:29.952 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 10 monclient: shutdown 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 -- 172.21.15.64:0/927615532 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 msgr2=0x7fa498146f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 --2- 172.21.15.64:0/927615532 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498146f90 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa490009e50 tx=0x7fa49002be40 comp rx=0 tx=0).stop 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 -- 172.21.15.64:0/927615532 shutdown_connections 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 --2- 172.21.15.64:0/927615532 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498146f90 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 --2- 172.21.15.64:0/927615532 >> v2:172.21.15.167:3300/0 conn(0x7fa4980b3270 0x7fa498146a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 -- 172.21.15.64:0/927615532 >> 172.21.15.64:0/927615532 conn(0x7fa4980b9030 msgr2=0x7fa4980b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 -- 172.21.15.64:0/927615532 shutdown_connections 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.950+0000 7fa4a01ce640 1 -- 172.21.15.64:0/927615532 wait complete. 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 10 monclient: build_initial_monmap 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 10 monclient: monmap: 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:29.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 1 librados: starting msgr at 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 1 librados: starting objecter 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 1 Processor -- start 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 1 -- start start 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 1 librados: setting wanted keys 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 1 librados: calling monclient init 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.951+0000 7fa4a01ce640 10 monclient: init 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa49806cdb0 0x7fa498141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): picked mon.noname-a con 0x7fa49806cdb0 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4980b3270 0x7fa498136f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): picked mon.noname-b con 0x7fa4980b3270 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): picked mon.noname-c con 0x7fa498140ed0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:29.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa498147650 con 0x7fa49806cdb0 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4981474d0 con 0x7fa4980b3270 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7fa49814dd80 con 0x7fa498140ed0 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.952+0000 7fa4a01ce640 10 monclient(hunting): authenticate will time out at 3059.111741s 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.956+0000 7fa49e76b640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.958+0000 7fa49d769640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4980b3270 0x7fa498136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49d769640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4980b3270 0x7fa498136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56694/0 (socket says 172.21.15.64:56694) 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49d769640 1 -- 172.21.15.64:0/2950479180 learned_addr learned my addr 172.21.15.64:0/2950479180 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49d769640 10 monclient(hunting): get_auth_request con 0x7fa4980b3270 auth_method 0 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49d769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49d769640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49d769640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.959+0000 7fa49df6a640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa49806cdb0 0x7fa498141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:29.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.960+0000 7fa49e76b640 10 monclient(hunting): get_auth_request con 0x7fa498140ed0 auth_method 0 2024-04-29T20:48:29.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.960+0000 7fa49e76b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.960+0000 7fa49e76b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.960+0000 7fa49e76b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49e76b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49e76b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49e76b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49df6a640 10 monclient(hunting): get_auth_request con 0x7fa49806cdb0 auth_method 0 2024-04-29T20:48:29.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49df6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:29.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49df6a640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:29.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.961+0000 7fa49df6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:29.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49df6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49df6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49df6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49d769640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49d769640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49d769640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 10 monclient(hunting): handle_auth_done global_id 16406 payload 824 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 1 -- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4980b3270 msgr2=0x7fa498136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4980b3270 0x7fa498136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 1 -- 172.21.15.64:0/2950479180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa49806cdb0 msgr2=0x7fa498141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa49806cdb0 0x7fa498141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 10 monclient: _finish_hunting 0 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 1 monclient: found mon.noname-c 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:29.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:48:29.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.965+0000 7fa49e76b640 1 -- 172.21.15.64:0/2950479180 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa498144090 con 0x7fa498140ed0 2024-04-29T20:48:29.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.969+0000 7fa49e76b640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498137460 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fa49806d4e0 tx=0x7fa490004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:29.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.969+0000 7fa48e7fc640 10 client.?.objecter ms_handle_connect 0x7fa498140ed0 2024-04-29T20:48:29.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:29.969+0000 7fa48e7fc640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:30.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490043030 con 0x7fa498140ed0 2024-04-29T20:48:30.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:30.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:30.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: dump: 2024-04-29T20:48:30.013 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:30.013 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: _finish_auth 0 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: _check_auth_tickets 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49003e450 con 0x7fa498140ed0 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49003e640 con 0x7fa498140ed0 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 5 monclient: authenticate success, global_id 16406 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 10 monclient: _renew_subs 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4981379a0 con 0x7fa498140ed0 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48e7fc640 10 monclient: dump: 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa48ffff640 4 set_mon_vals no callback set 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 10 client.16406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 10 monclient: _renew_subs 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa498142450 con 0x7fa498140ed0 2024-04-29T20:48:30.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.013+0000 7fa4a01ce640 1 librados: init done 2024-04-29T20:48:30.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.014+0000 7fa4a01ce640 10 client.16406.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:30.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.014+0000 7fa4a01ce640 15 client.16406.objecter _session_command_op_assign -1 1 2024-04-29T20:48:30.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.014+0000 7fa4a01ce640 20 client.16406.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:30.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.014+0000 7fa4a01ce640 10 client.16406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:30.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.014+0000 7fa4a01ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:30.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.014+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa498142f20 con 0x7fa498140ed0 2024-04-29T20:48:30.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.046+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa490046070 con 0x7fa498140ed0 2024-04-29T20:48:30.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.047+0000 7fa48e7fc640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa47c06e280 0x7fa47c070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.047+0000 7fa49df6a640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa47c06e280 0x7fa47c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.047+0000 7fa49df6a640 10 monclient: get_auth_request con 0x7fa47c06e280 auth_method 0 2024-04-29T20:48:30.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.047+0000 7fa48e7fc640 10 client.16406.objecter ms_handle_connect 0x7fa47c06e280 2024-04-29T20:48:30.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.047+0000 7fa49df6a640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa47c06e280 0x7fa47c070740 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa4940017f0 tx=0x7fa494009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:30.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7fa49007a560 con 0x7fa498140ed0 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 10 client.16406.objecter ms_dispatch 0x7fa498000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 3 client.16406.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 3 client.16406.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 10 client.16406.objecter checking command 1 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7fa47c075f80 0x7fa47c078420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 20 client.16406.objecter _get_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 15 client.16406.objecter _session_command_op_remove -1 1 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 20 client.16406.objecter _get_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 20 client.16406.objecter get_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 15 client.16406.objecter _session_command_op_assign 6 1 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 4 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 10 client.16406.objecter _send_command 1 2024-04-29T20:48:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 --> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa49000aea0 con 0x7fa47c075f80 2024-04-29T20:48:30.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 20 client.16406.objecter dump_active .. 0 homeless 2024-04-29T20:48:30.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7fa490078760 con 0x7fa498140ed0 2024-04-29T20:48:30.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.056+0000 7fa48e7fc640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:30.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.057+0000 7fa49d769640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7fa47c075f80 0x7fa47c078420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.057+0000 7fa49d769640 10 monclient: get_auth_request con 0x7fa47c075f80 auth_method 0 2024-04-29T20:48:30.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.060+0000 7fa48e7fc640 10 client.16406.objecter ms_handle_connect 0x7fa47c075f80 2024-04-29T20:48:30.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.060+0000 7fa49d769640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7fa47c075f80 0x7fa47c078420 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:30.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.084+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== osd.6 v2:172.21.15.167:6818/3916781155 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa49000aea0 con 0x7fa47c075f80 2024-04-29T20:48:30.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.084+0000 7fa48e7fc640 10 client.16406.objecter ms_dispatch 0x7fa498000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:30.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.084+0000 7fa48e7fc640 10 client.16406.objecter _finish_command 1 = system:0 2024-04-29T20:48:30.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.084+0000 7fa48e7fc640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 4 2024-04-29T20:48:30.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.084+0000 7fa48e7fc640 15 client.16406.objecter _session_command_op_remove 6 1 2024-04-29T20:48:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 10 client.16406.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 15 client.16406.objecter _session_command_op_assign -1 2 2024-04-29T20:48:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 20 client.16406.objecter _get_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 20 client.16406.objecter _get_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 15 client.16406.objecter _session_command_op_remove -1 2 2024-04-29T20:48:30.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 20 client.16406.objecter get_session s=0x7fa47c0758b0 osd=6 3 2024-04-29T20:48:30.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 15 client.16406.objecter _session_command_op_assign 6 2 2024-04-29T20:48:30.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 4 2024-04-29T20:48:30.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 10 client.16406.objecter _send_command 2 2024-04-29T20:48:30.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.100+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 --> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa49804fce0 con 0x7fa47c075f80 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa48e7fc640 1 -- 172.21.15.64:0/2950479180 <== osd.6 v2:172.21.15.167:6818/3916781155 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa49804fce0 con 0x7fa47c075f80 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa48e7fc640 10 client.16406.objecter ms_dispatch 0x7fa498000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa48e7fc640 10 client.16406.objecter _finish_command 2 = system:0 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa48e7fc640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 4 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa48e7fc640 15 client.16406.objecter _session_command_op_remove 6 2 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:30.120 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 10 librados: watch_flush enter 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 10 librados: watch_flush exit 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 10 client.16406.objecter close_session for osd.6 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7fa47c075f80 msgr2=0x7fa47c078420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6818/3916781155,v1:172.21.15.167:6819/3916781155] conn(0x7fa47c075f80 0x7fa47c078420 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 20 client.16406.objecter put_session s=0x7fa47c0758b0 osd=6 1 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 20 client.16406.objecter shutdown clearing up homeless session... 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 10 client.16406.objecter successfully canceled tick 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa47c06e280 msgr2=0x7fa47c070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7fa47c06e280 0x7fa47c070740 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa4940017f0 tx=0x7fa494009040 comp rx=0 tx=0).stop 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 10 monclient: shutdown 2024-04-29T20:48:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 msgr2=0x7fa498137460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 --2- 172.21.15.64:0/2950479180 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7fa498140ed0 0x7fa498137460 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fa49806d4e0 tx=0x7fa490004a40 comp rx=0 tx=0).stop 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa49e76b640 1 -- 172.21.15.64:0/2950479180 reap_dead start 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 shutdown_connections 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.120+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 >> 172.21.15.64:0/2950479180 conn(0x7fa4980b9030 msgr2=0x7fa4980b8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.121+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 shutdown_connections 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.121+0000 7fa4a01ce640 1 -- 172.21.15.64:0/2950479180 wait complete. 2024-04-29T20:48:30.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.121+0000 7fa4a01ce640 1 librados: shutdown 2024-04-29T20:48:30.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.236+0000 7f51b7315640 10 monclient: get_monmap_and_config 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.236+0000 7f51b7315640 10 monclient: build_initial_monmap 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.236+0000 7f51b7315640 10 monclient: monmap: 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:30.237142+0000 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:30.237142+0000 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:30.236 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:30.237 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:30.237 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:30.237 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.236+0000 7f51b7315640 1 Processor -- start 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 1 -- start start 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient: init 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f51b00b2230 0x7f51b0146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): picked mon.noname-c con 0x7f51b00b2230 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f51b00aa460 0x7f51b00b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): picked mon.noname-a con 0x7f51b00aa460 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51b00b5c60 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:30.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b0146eb0 con 0x7f51b00b5c60 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f51b00ac1f0 con 0x7f51b00aa460 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f51b00ac4d0 con 0x7f51b00b2230 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.237+0000 7f51b7315640 10 monclient(hunting): authenticate will time out at 3059.397333s 2024-04-29T20:48:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.239+0000 7f51b50b1640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f51b00b2230 0x7f51b0146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.240+0000 7f51b50b1640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f51b00b2230 0x7f51b0146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56702/0 (socket says 172.21.15.64:56702) 2024-04-29T20:48:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.240+0000 7f51b50b1640 1 -- 172.21.15.64:0/1341842624 learned_addr learned my addr 172.21.15.64:0/1341842624 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.240+0000 7f51b50b1640 10 monclient(hunting): get_auth_request con 0x7f51b00b2230 auth_method 0 2024-04-29T20:48:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.240+0000 7f51b50b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.240+0000 7f51b50b1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.240+0000 7f51b50b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.241+0000 7f51b50b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.241+0000 7f51b50b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.241+0000 7f51b50b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 10 monclient(hunting): handle_auth_done global_id 22201 payload 274 2024-04-29T20:48:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 1 -- 172.21.15.64:0/1341842624 >> v2:172.21.15.64:3300/0 conn(0x7f51b00aa460 msgr2=0x7f51b00b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 1 --2- 172.21.15.64:0/1341842624 >> v2:172.21.15.64:3300/0 conn(0x7f51b00aa460 0x7f51b00b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 1 -- 172.21.15.64:0/1341842624 >> v1:172.21.15.64:6789/0 conn(0x7f51b00b5c60 legacy=0x7f51b00b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 10 monclient: _finish_hunting 0 2024-04-29T20:48:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 1 monclient: found mon.noname-c 2024-04-29T20:48:30.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:30.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:30.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.243+0000 7f51b50b1640 1 -- 172.21.15.64:0/1341842624 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b00b3440 con 0x7f51b00b2230 2024-04-29T20:48:30.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.247+0000 7f51b50b1640 1 --2- 172.21.15.64:0/1341842624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00b2230 0x7f51b0146970 secure :-1 s=READY pgs=5640 cs=0 l=1 rev1=1 crypto rx=0x7f51b00b9df0 tx=0x7f51ac02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f14c652e57dfd497 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:30.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 1 -- 172.21.15.64:0/1341842624 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac02e070 con 0x7f51b00b2230 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: dump: 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:30.251 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: _finish_auth 0 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: _check_auth_tickets 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 1 -- 172.21.15.64:0/1341842624 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ac037070 con 0x7f51b00b2230 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 5 monclient: authenticate success, global_id 22201 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 1 -- 172.21.15.64:0/1341842624 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac031a00 con 0x7f51b00b2230 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 4 set_mon_vals no callback set 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51a7fff640 10 monclient: dump: 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:30.253 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 10 monclient: shutdown 2024-04-29T20:48:30.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:30.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 1 -- 172.21.15.64:0/1341842624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00b2230 msgr2=0x7f51b0146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 1 --2- 172.21.15.64:0/1341842624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00b2230 0x7f51b0146970 secure :-1 s=READY pgs=5640 cs=0 l=1 rev1=1 crypto rx=0x7f51b00b9df0 tx=0x7f51ac02b880 comp rx=0 tx=0).stop 2024-04-29T20:48:30.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 1 -- 172.21.15.64:0/1341842624 shutdown_connections 2024-04-29T20:48:30.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 1 --2- 172.21.15.64:0/1341842624 >> v2:172.21.15.64:3300/0 conn(0x7f51b00aa460 0x7f51b00b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 1 --2- 172.21.15.64:0/1341842624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00b2230 0x7f51b0146970 unknown :-1 s=CLOSED pgs=5640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.251+0000 7f51b7315640 1 -- 172.21.15.64:0/1341842624 >> 172.21.15.64:0/1341842624 conn(0x7f51b006c960 msgr2=0x7f51b006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.252+0000 7f51b7315640 1 -- 172.21.15.64:0/1341842624 shutdown_connections 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.252+0000 7f51b7315640 1 -- 172.21.15.64:0/1341842624 wait complete. 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.252+0000 7f51b7315640 10 monclient: build_initial_monmap 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.252+0000 7f51b7315640 10 monclient: monmap: 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:30.256 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 librados: starting msgr at 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 librados: starting objecter 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 Processor -- start 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 -- start start 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 librados: setting wanted keys 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 librados: calling monclient init 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient: init 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 0x7f51b00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): picked mon.noname-b con 0x7f51b00aa460 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f51b00b2230 0x7f51b0120e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): picked mon.noname-c con 0x7f51b00b2230 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51b00b5c60 0x7f51b01d8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): picked mon.noname-a con 0x7f51b00b5c60 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f51b00b2790 con 0x7f51b00b5c60 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f51b00b2610 con 0x7f51b00aa460 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f51b00b6090 con 0x7f51b00b2230 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.253+0000 7f51b7315640 10 monclient(hunting): authenticate will time out at 3059.413440s 2024-04-29T20:48:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.258+0000 7f51b48b0640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f51b00b2230 0x7f51b0120e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.260+0000 7f51b58b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51b00b5c60 0x7f51b01d8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 0x7f51b00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 0x7f51b00b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56718/0 (socket says 172.21.15.64:56718) 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 1 -- 172.21.15.64:0/3096888162 learned_addr learned my addr 172.21.15.64:0/3096888162 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 10 monclient(hunting): get_auth_request con 0x7f51b00aa460 auth_method 0 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b50b1640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b58b2640 10 monclient(hunting): get_auth_request con 0x7f51b00b5c60 auth_method 0 2024-04-29T20:48:30.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b58b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b58b2640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.262+0000 7f51b58b2640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.263+0000 7f51b50b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.263+0000 7f51b50b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.263+0000 7f51b50b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:30.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.266+0000 7f51b48b0640 10 monclient(hunting): get_auth_request con 0x7f51b00b2230 auth_method 0 2024-04-29T20:48:30.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.266+0000 7f51b48b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:30.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.266+0000 7f51b48b0640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:30.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.266+0000 7f51b48b0640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 10 monclient(hunting): handle_auth_done global_id 22204 payload 824 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 1 -- 172.21.15.64:0/3096888162 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f51b00b2230 msgr2=0x7f51b0120e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f51b00b2230 0x7f51b0120e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 1 -- 172.21.15.64:0/3096888162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51b00b5c60 msgr2=0x7f51b01d8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51b00b5c60 0x7f51b01d8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 10 monclient: _finish_hunting 0 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 1 monclient: found mon.noname-b 2024-04-29T20:48:30.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:30.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:30.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.267+0000 7f51b50b1640 1 -- 172.21.15.64:0/3096888162 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b00ae8f0 con 0x7f51b00aa460 2024-04-29T20:48:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.271+0000 7f51b50b1640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 0x7f51b00b1500 secure :-1 s=READY pgs=5641 cs=0 l=1 rev1=1 crypto rx=0x7f51ac02b8c0 tx=0x7f51ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.271+0000 7f51a5ffb640 10 client.?.objecter ms_handle_connect 0x7f51b00aa460 2024-04-29T20:48:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.271+0000 7f51a5ffb640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac032880 con 0x7f51b00aa460 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: dump: 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.285 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: _finish_auth 0 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: _check_auth_tickets 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ac032a20 con 0x7f51b00aa460 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac032e10 con 0x7f51b00aa460 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51b7315640 5 monclient: authenticate success, global_id 22204 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:30.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a5ffb640 10 monclient: dump: 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51a77fe640 4 set_mon_vals no callback set 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51b7315640 10 monclient: _renew_subs 2024-04-29T20:48:30.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51b7315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:30.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b0122650 con 0x7f51b00aa460 2024-04-29T20:48:30.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.284+0000 7f51b7315640 10 client.22204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:30.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 10 monclient: _renew_subs 2024-04-29T20:48:30.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:30.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b01d9170 con 0x7f51b00aa460 2024-04-29T20:48:30.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 1 librados: init done 2024-04-29T20:48:30.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:30.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:30.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:30.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.285+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b004fce0 con 0x7f51b00aa460 2024-04-29T20:48:30.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.618+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f51ac04c060 con 0x7f51b00aa460 2024-04-29T20:48:30.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.618+0000 7f51a5ffb640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f519406e2d0 0x7f5194070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.619+0000 7f51b48b0640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f519406e2d0 0x7f5194070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:30.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.619+0000 7f51b48b0640 10 monclient: get_auth_request con 0x7f519406e2d0 auth_method 0 2024-04-29T20:48:30.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.619+0000 7f51b48b0640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f519406e2d0 0x7f5194070790 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f51b00b0330 tx=0x7f51a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:30.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.619+0000 7f51a5ffb640 10 client.22204.objecter ms_handle_connect 0x7f519406e2d0 2024-04-29T20:48:30.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.624+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f51ac079830 con 0x7f51b00aa460 2024-04-29T20:48:30.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.624+0000 7f51a5ffb640 10 client.22204.objecter ms_dispatch 0x7f51b0000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:30.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.624+0000 7f51a5ffb640 3 client.22204.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:30.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.624+0000 7f51a5ffb640 3 client.22204.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:30.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.624+0000 7f51a5ffb640 20 client.22204.objecter dump_active .. 0 homeless 2024-04-29T20:48:30.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.634+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51ac040070 con 0x7f51b00aa460 2024-04-29T20:48:30.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.634+0000 7f51a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:30.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.634+0000 7f51a5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:30.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.845+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f51b00b9c80 con 0x7f519406e2d0 2024-04-29T20:48:30.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.845+0000 7f51a5ffb640 1 -- 172.21.15.64:0/3096888162 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f51b00b9c80 con 0x7f519406e2d0 2024-04-29T20:48:30.845 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 6 to deep-scrub 2024-04-29T20:48:30.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.847+0000 7f51b7315640 10 librados: watch_flush enter 2024-04-29T20:48:30.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 10 librados: watch_flush exit 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 20 client.22204.objecter shutdown clearing up homeless session... 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 10 client.22204.objecter successfully canceled tick 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f519406e2d0 msgr2=0x7f5194070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f519406e2d0 0x7f5194070790 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f51b00b0330 tx=0x7f51a0008040 comp rx=0 tx=0).stop 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 10 monclient: shutdown 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 msgr2=0x7f51b00b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 0x7f51b00b1500 secure :-1 s=READY pgs=5641 cs=0 l=1 rev1=1 crypto rx=0x7f51ac02b8c0 tx=0x7f51ac0046c0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 shutdown_connections 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f519406e2d0 0x7f5194070790 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f51b00b2230 0x7f51b0120e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51b00b5c60 0x7f51b01d8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 --2- 172.21.15.64:0/3096888162 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b00aa460 0x7f51b00b1500 unknown :-1 s=CLOSED pgs=5641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:30.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 >> 172.21.15.64:0/3096888162 conn(0x7f51b006c960 msgr2=0x7f51b00b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:30.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 shutdown_connections 2024-04-29T20:48:30.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 -- 172.21.15.64:0/3096888162 wait complete. 2024-04-29T20:48:30.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.848+0000 7f51b7315640 1 librados: shutdown 2024-04-29T20:48:30.859 INFO:tasks.ceph:Scrubbing osd.7 2024-04-29T20:48:30.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.991+0000 7f5fd88ca640 10 monclient: get_monmap_and_config 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.991+0000 7f5fd88ca640 10 monclient: build_initial_monmap 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.991+0000 7f5fd88ca640 10 monclient: monmap: 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:30.991950+0000 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:30.991950+0000 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:30.992 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.991+0000 7f5fd88ca640 1 Processor -- start 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.992+0000 7f5fd88ca640 1 -- start start 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.992+0000 7f5fd88ca640 10 monclient: init 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f5fd40b5b90 0x7f5fd4146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd40b5b90 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:30.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f5fd40b5f70 0x7f5fd40aa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd40b5f70 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b2d70 0x7f5fd40b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd40b2d70 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd40aa9a0 con 0x7f5fd40b5f70 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd40aabc0 con 0x7f5fd40b5b90 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd40b3220 con 0x7f5fd40b2d70 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:30.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.993+0000 7f5fd88ca640 10 monclient(hunting): authenticate will time out at 3060.152808s 2024-04-29T20:48:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:30.998+0000 7f5fd259c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f5fd40b5b90 0x7f5fd4146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.000+0000 7f5fd259c640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f5fd40b5b90 0x7f5fd4146900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56730/0 (socket says 172.21.15.64:56730) 2024-04-29T20:48:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.000+0000 7f5fd259c640 1 -- 172.21.15.64:0/273324960 learned_addr learned my addr 172.21.15.64:0/273324960 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.000+0000 7f5fd259c640 10 monclient(hunting): get_auth_request con 0x7f5fd40b5b90 auth_method 0 2024-04-29T20:48:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.000+0000 7f5fd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.000+0000 7f5fd259c640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.000+0000 7f5fd259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.001+0000 7f5fc9d9b640 1 --2- 172.21.15.64:0/273324960 >> v2:172.21.15.64:3300/0 conn(0x7f5fd40b5f70 0x7f5fd40aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.002+0000 7f5fc9d9b640 10 monclient(hunting): get_auth_request con 0x7f5fd40b5f70 auth_method 0 2024-04-29T20:48:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.002+0000 7f5fc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.002+0000 7f5fc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.002+0000 7f5fc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:31.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.003+0000 7f5fd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:31.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.003+0000 7f5fd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:31.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.003+0000 7f5fd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:31.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.004+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:31.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.004+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:31.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.004+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:31.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd2d9d640 1 --2- 172.21.15.64:0/273324960 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b2d70 0x7f5fd40b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 10 monclient(hunting): handle_auth_done global_id 22207 payload 274 2024-04-29T20:48:31.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 1 -- 172.21.15.64:0/273324960 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b2d70 msgr2=0x7f5fd40b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 1 --2- 172.21.15.64:0/273324960 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b2d70 0x7f5fd40b1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 1 -- 172.21.15.64:0/273324960 >> v2:172.21.15.64:3300/0 conn(0x7f5fd40b5f70 msgr2=0x7f5fd40aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 1 --2- 172.21.15.64:0/273324960 >> v2:172.21.15.64:3300/0 conn(0x7f5fd40b5f70 0x7f5fd40aa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 10 monclient: _finish_hunting 0 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 1 monclient: found mon.noname-c 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.006+0000 7f5fd259c640 1 -- 172.21.15.64:0/273324960 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd40b5450 con 0x7f5fd40b5b90 2024-04-29T20:48:31.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.016+0000 7f5fd259c640 1 --2- 172.21.15.64:0/273324960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd40b5b90 0x7f5fd4146900 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40b9d80 tx=0x7f5fc402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b7ffd5831b94ca server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 1 -- 172.21.15.64:0/273324960 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4035020 con 0x7f5fd40b5b90 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: dump: 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: _finish_auth 0 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: _check_auth_tickets 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 1 -- 172.21.15.64:0/273324960 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc4031870 con 0x7f5fd40b5b90 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 1 -- 172.21.15.64:0/273324960 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4031c40 con 0x7f5fd40b5b90 2024-04-29T20:48:31.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd88ca640 5 monclient: authenticate success, global_id 22207 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd88ca640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd88ca640 4 set_mon_vals no callback set 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.079+0000 7f5fd1d9b640 10 monclient: dump: 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.081 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.082 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 10 monclient: shutdown 2024-04-29T20:48:31.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:31.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 -- 172.21.15.64:0/273324960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd40b5b90 msgr2=0x7f5fd4146900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/273324960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd40b5b90 0x7f5fd4146900 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40b9d80 tx=0x7f5fc402b910 comp rx=0 tx=0).stop 2024-04-29T20:48:31.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 -- 172.21.15.64:0/273324960 shutdown_connections 2024-04-29T20:48:31.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/273324960 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b2d70 0x7f5fd40b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/273324960 >> v2:172.21.15.64:3300/0 conn(0x7f5fd40b5f70 0x7f5fd40aa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/273324960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd40b5b90 0x7f5fd4146900 unknown :-1 s=CLOSED pgs=5642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 -- 172.21.15.64:0/273324960 >> 172.21.15.64:0/273324960 conn(0x7f5fd406c9d0 msgr2=0x7f5fd406cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 -- 172.21.15.64:0/273324960 shutdown_connections 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.080+0000 7f5fd88ca640 1 -- 172.21.15.64:0/273324960 wait complete. 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 10 monclient: build_initial_monmap 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 10 monclient: monmap: 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:31.085 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 1 librados: starting msgr at 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 1 librados: starting objecter 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 1 Processor -- start 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 1 -- start start 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 1 librados: setting wanted keys 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 1 librados: calling monclient init 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.081+0000 7f5fd88ca640 10 monclient: init 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-29T20:48:31.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fd40b2d70 0x7f5fd413d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd40b2d70 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b5b90 0x7f5fd40afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd40b5b90 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd41d8a30 0x7f5fd40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd41d8a30 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd40b33d0 con 0x7f5fd40b2d70 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd40b25b0 con 0x7f5fd41d8a30 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd4143400 con 0x7f5fd40b5b90 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fd88ca640 10 monclient(hunting): authenticate will time out at 3060.241738s 2024-04-29T20:48:31.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.082+0000 7f5fc9d9b640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b5b90 0x7f5fd40afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b5b90 0x7f5fd40afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:45804/0 (socket says 172.21.15.64:45804) 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 1 -- 172.21.15.64:0/1951399811 learned_addr learned my addr 172.21.15.64:0/1951399811 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): get_auth_request con 0x7f5fd40b5b90 auth_method 0 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:31.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.083+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 10 monclient(hunting): handle_auth_done global_id 16412 payload 824 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 1 -- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd41d8a30 msgr2=0x7f5fd40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fd41d8a30 0x7f5fd40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 1 -- 172.21.15.64:0/1951399811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fd40b2d70 msgr2=0x7f5fd413d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fd40b2d70 0x7f5fd413d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 10 monclient: _finish_hunting 0 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 1 monclient: found mon.noname-c 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:31.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.64:3301/0 2024-04-29T20:48:31.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.086+0000 7f5fc9d9b640 1 -- 172.21.15.64:0/1951399811 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd41209c0 con 0x7f5fd40b5b90 2024-04-29T20:48:31.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.088+0000 7f5fc9d9b640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b5b90 0x7f5fd40afa70 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00e8d0 tx=0x7f5fbc00ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.088+0000 7f5fcb7fe640 10 client.?.objecter ms_handle_connect 0x7f5fd40b5b90 2024-04-29T20:48:31.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.088+0000 7f5fcb7fe640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:31.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc017070 con 0x7f5fd40b5b90 2024-04-29T20:48:31.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:31.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: dump: 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: _finish_auth 0 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: _check_auth_tickets 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fbc00c940 con 0x7f5fd40b5b90 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 5 monclient: authenticate success, global_id 16412 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 10 monclient: _renew_subs 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd40b0810 con 0x7f5fd40b5b90 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc00cd30 con 0x7f5fd40b5b90 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fcb7fe640 10 monclient: dump: 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.094 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd159a640 4 set_mon_vals no callback set 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 10 client.16412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 10 monclient: _renew_subs 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd41d90a0 con 0x7f5fd40b5b90 2024-04-29T20:48:31.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.092+0000 7f5fd88ca640 1 librados: init done 2024-04-29T20:48:31.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.093+0000 7f5fd88ca640 10 client.16412.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:31.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.093+0000 7f5fd88ca640 15 client.16412.objecter _session_command_op_assign -1 1 2024-04-29T20:48:31.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.093+0000 7f5fd88ca640 20 client.16412.objecter _get_session osd=-1 returning homeless 2024-04-29T20:48:31.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.093+0000 7f5fd88ca640 10 client.16412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:31.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.093+0000 7f5fd88ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.64:3301/0 2024-04-29T20:48:31.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.093+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5fd404fce0 con 0x7f5fd40b5b90 2024-04-29T20:48:31.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.113+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== mon.2 v2:172.21.15.64:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5fbc01a070 con 0x7f5fd40b5b90 2024-04-29T20:48:31.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.113+0000 7f5fcb7fe640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5fb406e2d0 0x7f5fb4070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.114+0000 7f5fd259c640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5fb406e2d0 0x7f5fb4070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.114+0000 7f5fd259c640 10 monclient: get_auth_request con 0x7f5fb406e2d0 auth_method 0 2024-04-29T20:48:31.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.114+0000 7f5fcb7fe640 10 client.16412.objecter ms_handle_connect 0x7f5fb406e2d0 2024-04-29T20:48:31.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.114+0000 7f5fd259c640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5fb406e2d0 0x7f5fb4070790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4036000 tx=0x7f5fc4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== mon.2 v2:172.21.15.64:3301/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f5fbc057ca0 con 0x7f5fd40b5b90 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 10 client.16412.objecter ms_dispatch 0x7f5fd4000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 3 client.16412.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 3 client.16412.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 10 client.16412.objecter checking command 1 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f5fb4075fd0 0x7f5fb4078470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 20 client.16412.objecter _get_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 15 client.16412.objecter _session_command_op_remove -1 1 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 20 client.16412.objecter _get_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 20 client.16412.objecter get_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 15 client.16412.objecter _session_command_op_assign 7 1 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 4 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 10 client.16412.objecter _send_command 1 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 --> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fbc00ae90 con 0x7f5fb4075fd0 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 20 client.16412.objecter dump_active .. 0 homeless 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== mon.2 v2:172.21.15.64:3301/0 6 ==== mon_get_version_reply(handle=1 version=976) ==== 24+0+0 (secure 0 0 0) 0x7f5fbc055ea0 con 0x7f5fd40b5b90 2024-04-29T20:48:31.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.117+0000 7f5fcb7fe640 10 monclient: handle_get_version_reply finishing 1 version 976 2024-04-29T20:48:31.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.118+0000 7f5fd2d9d640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f5fb4075fd0 0x7f5fb4078470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.118+0000 7f5fd2d9d640 10 monclient: get_auth_request con 0x7f5fb4075fd0 auth_method 0 2024-04-29T20:48:31.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.121+0000 7f5fd2d9d640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f5fb4075fd0 0x7f5fb4078470 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.121+0000 7f5fcb7fe640 10 client.16412.objecter ms_handle_connect 0x7f5fb4075fd0 2024-04-29T20:48:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.145+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== osd.7 v2:172.21.15.167:6810/2258625350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5fbc00ae90 con 0x7f5fb4075fd0 2024-04-29T20:48:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.145+0000 7f5fcb7fe640 10 client.16412.objecter ms_dispatch 0x7f5fd4000b90 command_reply(tid 1: 0 ) 2024-04-29T20:48:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.145+0000 7f5fcb7fe640 10 client.16412.objecter _finish_command 1 = system:0 2024-04-29T20:48:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.145+0000 7f5fcb7fe640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 4 2024-04-29T20:48:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.145+0000 7f5fcb7fe640 15 client.16412.objecter _session_command_op_remove 7 1 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 10 client.16412.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 15 client.16412.objecter _session_command_op_assign -1 2 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 20 client.16412.objecter _get_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 20 client.16412.objecter _get_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 15 client.16412.objecter _session_command_op_remove -1 2 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 20 client.16412.objecter get_session s=0x7f5fb4075900 osd=7 3 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 15 client.16412.objecter _session_command_op_assign 7 2 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 4 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 10 client.16412.objecter _send_command 2 2024-04-29T20:48:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.159+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 --> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5fd40b77d0 con 0x7f5fb4075fd0 2024-04-29T20:48:31.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fcb7fe640 1 -- 172.21.15.64:0/1951399811 <== osd.7 v2:172.21.15.167:6810/2258625350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5fd40b77d0 con 0x7f5fb4075fd0 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fcb7fe640 10 client.16412.objecter ms_dispatch 0x7f5fd4000b90 command_reply(tid 2: 0 ) 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fcb7fe640 10 client.16412.objecter _finish_command 2 = system:0 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fcb7fe640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 4 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fcb7fe640 15 client.16412.objecter _session_command_op_remove 7 2 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-29T20:48:31.173 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-29T20:48:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 10 librados: watch_flush enter 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 10 librados: watch_flush exit 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 10 client.16412.objecter close_session for osd.7 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f5fb4075fd0 msgr2=0x7f5fb4078470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6810/2258625350,v1:172.21.15.167:6811/2258625350] conn(0x7f5fb4075fd0 0x7f5fb4078470 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 20 client.16412.objecter put_session s=0x7f5fb4075900 osd=7 1 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 20 client.16412.objecter shutdown clearing up homeless session... 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 10 client.16412.objecter successfully canceled tick 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5fb406e2d0 msgr2=0x7f5fb4070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f5fb406e2d0 0x7f5fb4070790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4036000 tx=0x7f5fc4030000 comp rx=0 tx=0).stop 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 10 monclient: shutdown 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b5b90 msgr2=0x7f5fd40afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 --2- 172.21.15.64:0/1951399811 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f5fd40b5b90 0x7f5fd40afa70 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00e8d0 tx=0x7f5fbc00ed90 comp rx=0 tx=0).stop 2024-04-29T20:48:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd2d9d640 1 -- 172.21.15.64:0/1951399811 reap_dead start 2024-04-29T20:48:31.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 shutdown_connections 2024-04-29T20:48:31.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 >> 172.21.15.64:0/1951399811 conn(0x7f5fd406c9d0 msgr2=0x7f5fd40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:31.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 shutdown_connections 2024-04-29T20:48:31.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 -- 172.21.15.64:0/1951399811 wait complete. 2024-04-29T20:48:31.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.173+0000 7f5fd88ca640 1 librados: shutdown 2024-04-29T20:48:31.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-29T20:48:31.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.292+0000 7f3a267b2640 10 monclient: get_monmap_and_config 2024-04-29T20:48:31.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.292+0000 7f3a267b2640 10 monclient: build_initial_monmap 2024-04-29T20:48:31.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.292+0000 7f3a267b2640 10 monclient: monmap: 2024-04-29T20:48:31.292 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:31.293510+0000 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:31.293510+0000 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:31.293 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.293+0000 7f3a267b2640 1 Processor -- start 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.293+0000 7f3a267b2640 1 -- start start 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.293+0000 7f3a267b2640 10 monclient: init 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.293+0000 7f3a267b2640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.293+0000 7f3a267b2640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a200b22a0 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a20146a40 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3a200b2c20 0x7f3a2014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 10 monclient(hunting): picked mon.noname-a con 0x7f3a200b2c20 addr v2:172.21.15.64:3300/0 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a200b30b0 con 0x7f3a20146a40 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a20146e60 con 0x7f3a200b22a0 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 -- 0x7f3a200b33d0 con 0x7f3a200b2c20 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a267b2640 10 monclient(hunting): authenticate will time out at 3060.453633s 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a24d4f640 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3a200b2c20 0x7f3a2014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a1ffff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f3a200b22a0 0x7f3a200b5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:60204/0 (socket says 172.21.15.64:60204) 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.294+0000 7f3a1ffff640 1 -- 172.21.15.64:0/2418060130 learned_addr learned my addr 172.21.15.64:0/2418060130 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): get_auth_request con 0x7f3a200b2c20 auth_method 0 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:31.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:31.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:31.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:31.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.295+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:31.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_done global_id 14691 payload 274 2024-04-29T20:48:31.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 1 -- 172.21.15.64:0/2418060130 >> v1:172.21.15.167:6789/0 conn(0x7f3a200b22a0 legacy=0x7f3a200b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 1 -- 172.21.15.64:0/2418060130 >> v1:172.21.15.64:6789/0 conn(0x7f3a20146a40 legacy=0x7f3a200aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 10 monclient: _finish_hunting 0 2024-04-29T20:48:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 1 monclient: found mon.noname-a 2024-04-29T20:48:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.64:3300/0 2024-04-29T20:48:31.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.297+0000 7f3a24d4f640 1 -- 172.21.15.64:0/2418060130 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a200ac1f0 con 0x7f3a200b2c20 2024-04-29T20:48:31.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.301+0000 7f3a24d4f640 1 --2- 172.21.15.64:0/2418060130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 0x7f3a2014ede0 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009d90 tx=0x7f3a1802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ebef14bb8cef5d server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 1 -- 172.21.15.64:0/2418060130 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18039070 con 0x7f3a200b2c20 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: dump: 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: _finish_auth 0 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 1 -- 172.21.15.64:0/2418060130 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1802cc60 con 0x7f3a200b2c20 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 1 -- 172.21.15.64:0/2418060130 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18034660 con 0x7f3a200b2c20 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 5 monclient: authenticate success, global_id 14691 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 4 set_mon_vals no callback set 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a1effd640 10 monclient: dump: 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.355 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 10 monclient: shutdown 2024-04-29T20:48:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 1 -- 172.21.15.64:0/2418060130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 msgr2=0x7f3a2014ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 1 --2- 172.21.15.64:0/2418060130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 0x7f3a2014ede0 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009d90 tx=0x7f3a1802beb0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 1 -- 172.21.15.64:0/2418060130 shutdown_connections 2024-04-29T20:48:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 1 --2- 172.21.15.64:0/2418060130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 0x7f3a2014ede0 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 1 -- 172.21.15.64:0/2418060130 >> 172.21.15.64:0/2418060130 conn(0x7f3a2006c940 msgr2=0x7f3a2006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.353+0000 7f3a267b2640 1 -- 172.21.15.64:0/2418060130 shutdown_connections 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.354+0000 7f3a267b2640 1 -- 172.21.15.64:0/2418060130 wait complete. 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.354+0000 7f3a267b2640 10 monclient: build_initial_monmap 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.354+0000 7f3a267b2640 10 monclient: monmap: 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:31.359 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.354+0000 7f3a267b2640 1 librados: starting msgr at 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.354+0000 7f3a267b2640 1 librados: starting objecter 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.354+0000 7f3a267b2640 1 Processor -- start 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.355+0000 7f3a267b2640 1 -- start start 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.355+0000 7f3a267b2640 1 librados: setting wanted keys 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.355+0000 7f3a267b2640 1 librados: calling monclient init 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.355+0000 7f3a267b2640 10 monclient: init 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3a200b22a0 0x7f3a2013e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): picked mon.noname-c con 0x7f3a200b22a0 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:31.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 0x7f3a200af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): picked mon.noname-a con 0x7f3a200b2c20 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 0x7f3a200ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): picked mon.noname-b con 0x7f3a20146a40 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3a200b2800 con 0x7f3a200b2c20 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3a200b2680 con 0x7f3a20146a40 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2014f320 con 0x7f3a200b22a0 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a267b2640 10 monclient(hunting): authenticate will time out at 3060.515682s 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.356+0000 7f3a24d4f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 0x7f3a200ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 0x7f3a200ad980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56744/0 (socket says 172.21.15.64:56744) 2024-04-29T20:48:31.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 1 -- 172.21.15.64:0/79756298 learned_addr learned my addr 172.21.15.64:0/79756298 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): get_auth_request con 0x7f3a20146a40 auth_method 0 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a1ffff640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3a200b22a0 0x7f3a2013e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request con 0x7f3a200b22a0 auth_method 0 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a1ffff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.357+0000 7f3a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.358+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.358+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.358+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 10 monclient(hunting): handle_auth_done global_id 22210 payload 824 2024-04-29T20:48:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 1 -- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3a200b22a0 msgr2=0x7f3a2013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3a200b22a0 0x7f3a2013e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 1 -- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 msgr2=0x7f3a200af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 0x7f3a200af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 10 monclient: _finish_hunting 0 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 1 monclient: found mon.noname-b 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.360+0000 7f3a24d4f640 1 -- 172.21.15.64:0/79756298 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a200acbe0 con 0x7f3a20146a40 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.362+0000 7f3a24d4f640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 0x7f3a200ad980 secure :-1 s=READY pgs=5644 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009d90 tx=0x7f3a18004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.362+0000 7f3a1cff9640 10 client.?.objecter ms_handle_connect 0x7f3a20146a40 2024-04-29T20:48:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.362+0000 7f3a1cff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18039050 con 0x7f3a20146a40 2024-04-29T20:48:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: dump: 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: _finish_auth 0 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: _check_auth_tickets 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1803a040 con 0x7f3a20146a40 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1803e610 con 0x7f3a20146a40 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1cff9640 10 monclient: dump: 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 5 monclient: authenticate success, global_id 22210 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 10 monclient: _renew_subs 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a200aec80 con 0x7f3a20146a40 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a1e7fc640 4 set_mon_vals no callback set 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 10 client.22210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 10 monclient: _renew_subs 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a200ace80 con 0x7f3a20146a40 2024-04-29T20:48:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.366+0000 7f3a267b2640 1 librados: init done 2024-04-29T20:48:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.367+0000 7f3a267b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.367+0000 7f3a267b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.367+0000 7f3a267b2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.367+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2004fce0 con 0x7f3a20146a40 2024-04-29T20:48:31.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.388+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a18047020 con 0x7f3a20146a40 2024-04-29T20:48:31.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.389+0000 7f3a1cff9640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3a0006e280 0x7f3a00070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:31.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.389+0000 7f3a1ffff640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3a0006e280 0x7f3a00070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:31.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.389+0000 7f3a1ffff640 10 monclient: get_auth_request con 0x7f3a0006e280 auth_method 0 2024-04-29T20:48:31.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.390+0000 7f3a1cff9640 10 client.22210.objecter ms_handle_connect 0x7f3a0006e280 2024-04-29T20:48:31.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.390+0000 7f3a1ffff640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3a0006e280 0x7f3a00070740 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3a08007fd0 tx=0x7f3a08007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:31.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.690+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f3a1807a790 con 0x7f3a20146a40 2024-04-29T20:48:31.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.690+0000 7f3a1cff9640 10 client.22210.objecter ms_dispatch 0x7f3a20000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:31.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.690+0000 7f3a1cff9640 3 client.22210.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:31.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.690+0000 7f3a1cff9640 3 client.22210.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:31.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.691+0000 7f3a1cff9640 20 client.22210.objecter dump_active .. 0 homeless 2024-04-29T20:48:31.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.701+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a1804e020 con 0x7f3a20146a40 2024-04-29T20:48:31.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.701+0000 7f3a1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:31.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.701+0000 7f3a1cff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:31.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.898+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f3a2014ed50 con 0x7f3a0006e280 2024-04-29T20:48:31.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.899+0000 7f3a1cff9640 1 -- 172.21.15.64:0/79756298 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3a2014ed50 con 0x7f3a0006e280 2024-04-29T20:48:31.899 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 7 to deep-scrub 2024-04-29T20:48:31.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 10 librados: watch_flush enter 2024-04-29T20:48:31.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 10 librados: watch_flush exit 2024-04-29T20:48:31.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 20 client.22210.objecter shutdown clearing up homeless session... 2024-04-29T20:48:31.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 10 client.22210.objecter successfully canceled tick 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3a0006e280 msgr2=0x7f3a00070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3a0006e280 0x7f3a00070740 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3a08007fd0 tx=0x7f3a08007e80 comp rx=0 tx=0).stop 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 10 monclient: shutdown 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 msgr2=0x7f3a200ad980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 0x7f3a200ad980 secure :-1 s=READY pgs=5644 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009d90 tx=0x7f3a18004a40 comp rx=0 tx=0).stop 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 shutdown_connections 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f3a0006e280 0x7f3a00070740 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a20146a40 0x7f3a200ad980 unknown :-1 s=CLOSED pgs=5644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3a200b2c20 0x7f3a200af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 --2- 172.21.15.64:0/79756298 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f3a200b22a0 0x7f3a2013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 >> 172.21.15.64:0/79756298 conn(0x7f3a2006c940 msgr2=0x7f3a200aa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 shutdown_connections 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 -- 172.21.15.64:0/79756298 wait complete. 2024-04-29T20:48:31.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:31.902+0000 7f3a267b2640 1 librados: shutdown 2024-04-29T20:48:31.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.030+0000 7f1fc9e33640 10 monclient: get_monmap_and_config 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.030+0000 7f1fc9e33640 10 monclient: build_initial_monmap 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.030+0000 7f1fc9e33640 10 monclient: monmap: 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:32.031101+0000 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:32.031101+0000 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:32.030 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:32.031 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:32.031 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:32.031 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:32.031 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:32.031 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:32.031 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:32.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.030+0000 7f1fc9e33640 1 Processor -- start 2024-04-29T20:48:32.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 1 -- start start 2024-04-29T20:48:32.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient: init 2024-04-29T20:48:32.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fc40b9570 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc40b3240 0x7f1fc4140e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc40b3240 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1fc4146a90 0x7f1fc4146f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc4146a90 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc40abd80 con 0x7f1fc40b9570 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc40b9950 con 0x7f1fc4146a90 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc40aac70 con 0x7f1fc40b3240 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.031+0000 7f1fc9e33640 10 monclient(hunting): authenticate will time out at 3061.191241s 2024-04-29T20:48:32.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc3fff640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f1fc4146a90 0x7f1fc4146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc37fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1fc40b9570 0x7f1fc4140720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60548/0 (socket says 172.21.15.64:60548) 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc37fe640 1 -- 172.21.15.64:0/2668923951 learned_addr learned my addr 172.21.15.64:0/2668923951 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc3fff640 10 monclient(hunting): get_auth_request con 0x7f1fc4146a90 auth_method 0 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.032+0000 7f1fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.033+0000 7f1fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.033+0000 7f1fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:32.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.033+0000 7f1fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 10 monclient(hunting): handle_auth_done global_id 22213 payload 274 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 1 -- 172.21.15.64:0/2668923951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc40b3240 msgr2=0x7f1fc4140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 1 --2- 172.21.15.64:0/2668923951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc40b3240 0x7f1fc4140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 1 -- 172.21.15.64:0/2668923951 >> v1:172.21.15.64:6789/0 conn(0x7f1fc40b9570 legacy=0x7f1fc4140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 10 monclient: _finish_hunting 0 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 1 monclient: found mon.noname-c 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:32.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.034+0000 7f1fc3fff640 1 -- 172.21.15.64:0/2668923951 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc40abff0 con 0x7f1fc4146a90 2024-04-29T20:48:32.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.036+0000 7f1fc3fff640 1 --2- 172.21.15.64:0/2668923951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc4146a90 0x7f1fc4146f40 secure :-1 s=READY pgs=5645 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009dc0 tx=0x7f1fbc02be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d1dab03f94ef35 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:32.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 1 -- 172.21.15.64:0/2668923951 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc039070 con 0x7f1fc4146a90 2024-04-29T20:48:32.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: dump: 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: _finish_auth 0 2024-04-29T20:48:32.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: _check_auth_tickets 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 1 -- 172.21.15.64:0/2668923951 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fbc02cc40 con 0x7f1fc4146a90 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 5 monclient: authenticate success, global_id 22213 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 1 -- 172.21.15.64:0/2668923951 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc034440 con 0x7f1fc4146a90 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc27fc640 10 monclient: dump: 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:32.041 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:32.042 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:32.042 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:32.042 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:32.042 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:32.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:32.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 4 set_mon_vals no callback set 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 10 monclient: shutdown 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 -- 172.21.15.64:0/2668923951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc4146a90 msgr2=0x7f1fc4146f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/2668923951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc4146a90 0x7f1fc4146f40 secure :-1 s=READY pgs=5645 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009dc0 tx=0x7f1fbc02be90 comp rx=0 tx=0).stop 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 -- 172.21.15.64:0/2668923951 shutdown_connections 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/2668923951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc4146a90 0x7f1fc4146f40 unknown :-1 s=CLOSED pgs=5645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/2668923951 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc40b3240 0x7f1fc4140e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 -- 172.21.15.64:0/2668923951 >> 172.21.15.64:0/2668923951 conn(0x7f1fc406cba0 msgr2=0x7f1fc406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 -- 172.21.15.64:0/2668923951 shutdown_connections 2024-04-29T20:48:32.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.040+0000 7f1fc9e33640 1 -- 172.21.15.64:0/2668923951 wait complete. 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.041+0000 7f1fc9e33640 10 monclient: build_initial_monmap 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.041+0000 7f1fc9e33640 10 monclient: monmap: 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:32.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.041+0000 7f1fc9e33640 1 librados: starting msgr at 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.041+0000 7f1fc9e33640 1 librados: starting objecter 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.041+0000 7f1fc9e33640 1 Processor -- start 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 -- start start 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 librados: setting wanted keys 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 librados: calling monclient init 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient: init 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc40b3240 0x7f1fc4142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc40b3240 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 0x7f1fc41dc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc40b9570 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc4146a90 0x7f1fc41dc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc4146a90 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:32.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc4147480 con 0x7f1fc40b3240 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc40ac390 con 0x7f1fc40b9570 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc4147600 con 0x7f1fc4146a90 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.042+0000 7f1fc9e33640 10 monclient(hunting): authenticate will time out at 3061.202284s 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 0x7f1fc41dc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc40b3240 0x7f1fc4142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 0x7f1fc41dc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.64:56760/0 (socket says 172.21.15.64:56760) 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 1 -- 172.21.15.64:0/3382965868 learned_addr learned my addr 172.21.15.64:0/3382965868 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f1fc40b9570 auth_method 0 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:32.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc37fe640 10 monclient(hunting): get_auth_request con 0x7f1fc40b3240 auth_method 0 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc37fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.043+0000 7f1fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.044+0000 7f1fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.044+0000 7f1fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.044+0000 7f1fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.044+0000 7f1fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.044+0000 7f1fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:32.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.044+0000 7f1fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:32.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 10 monclient(hunting): handle_auth_done global_id 22216 payload 824 2024-04-29T20:48:32.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 1 -- 172.21.15.64:0/3382965868 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc4146a90 msgr2=0x7f1fc41dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:32.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc4146a90 0x7f1fc41dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 1 -- 172.21.15.64:0/3382965868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc40b3240 msgr2=0x7f1fc4142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc40b3240 0x7f1fc4142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 10 monclient: _finish_hunting 0 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 1 monclient: found mon.noname-b 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.046+0000 7f1fc2ffd640 1 -- 172.21.15.64:0/3382965868 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc413af20 con 0x7f1fc40b9570 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.048+0000 7f1fc2ffd640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 0x7f1fc41dc200 secure :-1 s=READY pgs=5646 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400b450 tx=0x7f1fb400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.048+0000 7f1fabfff640 10 client.?.objecter ms_handle_connect 0x7f1fc40b9570 2024-04-29T20:48:32.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.048+0000 7f1fabfff640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:32.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4015020 con 0x7f1fc40b9570 2024-04-29T20:48:32.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:32.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:32.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: dump: 2024-04-29T20:48:32.052 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: _finish_auth 0 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: _check_auth_tickets 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fb4011830 con 0x7f1fc40b9570 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 5 monclient: authenticate success, global_id 22216 2024-04-29T20:48:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 10 monclient: _renew_subs 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc413b200 con 0x7f1fc40b9570 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4011c20 con 0x7f1fc40b9570 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fabfff640 10 monclient: dump: 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:32.054 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc1ffb640 4 set_mon_vals no callback set 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 10 client.22216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 10 monclient: _renew_subs 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc41429c0 con 0x7f1fc40b9570 2024-04-29T20:48:32.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.052+0000 7f1fc9e33640 1 librados: init done 2024-04-29T20:48:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.053+0000 7f1fc9e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.053+0000 7f1fc9e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.053+0000 7f1fc9e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.053+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc40b76f0 con 0x7f1fc40b9570 2024-04-29T20:48:32.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.073+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1fb4018020 con 0x7f1fc40b9570 2024-04-29T20:48:32.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.073+0000 7f1fabfff640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1fb006e300 0x7f1fb00707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:32.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.073+0000 7f1fc37fe640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1fb006e300 0x7f1fb00707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:32.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.073+0000 7f1fc37fe640 10 monclient: get_auth_request con 0x7f1fb006e300 auth_method 0 2024-04-29T20:48:32.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.074+0000 7f1fabfff640 10 client.22216.objecter ms_handle_connect 0x7f1fb006e300 2024-04-29T20:48:32.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.074+0000 7f1fc37fe640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1fb006e300 0x7f1fb00707c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8007fd0 tx=0x7f1fb8007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:32.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.079+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f1fb4057da0 con 0x7f1fc40b9570 2024-04-29T20:48:32.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.079+0000 7f1fabfff640 10 client.22216.objecter ms_dispatch 0x7f1fc4000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:32.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.079+0000 7f1fabfff640 3 client.22216.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:32.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.079+0000 7f1fabfff640 3 client.22216.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:32.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.079+0000 7f1fabfff640 20 client.22216.objecter dump_active .. 0 homeless 2024-04-29T20:48:32.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.342+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1fc40b76f0 con 0x7f1fc40b9570 2024-04-29T20:48:32.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.342+0000 7f1fabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:32.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.342+0000 7f1fabfff640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:32.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.576+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1fc404fce0 con 0x7f1fb006e300 2024-04-29T20:48:32.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.580+0000 7f1fabfff640 1 -- 172.21.15.64:0/3382965868 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+115822 (secure 0 0 0) 0x7f1fc404fce0 con 0x7f1fb006e300 2024-04-29T20:48:32.581 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:48:32.581 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 10 librados: watch_flush enter 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 10 librados: watch_flush exit 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 20 client.22216.objecter shutdown clearing up homeless session... 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 10 client.22216.objecter successfully canceled tick 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1fb006e300 msgr2=0x7f1fb00707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1fb006e300 0x7f1fb00707c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8007fd0 tx=0x7f1fb8007e80 comp rx=0 tx=0).stop 2024-04-29T20:48:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 10 monclient: shutdown 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 msgr2=0x7f1fc41dc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 0x7f1fc41dc200 secure :-1 s=READY pgs=5646 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400b450 tx=0x7f1fb400b910 comp rx=0 tx=0).stop 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 shutdown_connections 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f1fb006e300 0x7f1fb00707c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f1fc4146a90 0x7f1fc41dc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc40b9570 0x7f1fc41dc200 unknown :-1 s=CLOSED pgs=5646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 --2- 172.21.15.64:0/3382965868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc40b3240 0x7f1fc4142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 >> 172.21.15.64:0/3382965868 conn(0x7f1fc406cba0 msgr2=0x7f1fc40b69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 shutdown_connections 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 -- 172.21.15.64:0/3382965868 wait complete. 2024-04-29T20:48:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:32.582+0000 7f1fc9e33640 1 librados: shutdown 2024-04-29T20:48:32.593 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1756,"stamp":"2024-04-29T20:48:31.632445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":616068,"kb_used_data":44600,"kb_used_omap":12,"kb_used_meta":571379,"kb_avail":754358652,"statfs":{"total":773094113280,"available":772463259648,"internally_reserved":0,"allocated":45670400,"data_stored":23569882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":585092658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":119,"apply_latency_ms":119,"commit_latency_ns":119000000,"apply_latency_ns":119000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001902"},"pg_stats":[{"pgid":"11.1b","version":"0'0","reported_seq":1869,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.342588+0000","last_change":"2024-04-29T20:48:28.343429+0000","last_active":"2024-04-29T20:48:28.342588+0000","last_peered":"2024-04-29T20:48:28.342588+0000","last_clean":"2024-04-29T20:48:28.342588+0000","last_became_active":"2024-04-29T20:25:19.211321+0000","last_became_peered":"2024-04-29T20:25:19.211321+0000","last_unstale":"2024-04-29T20:48:28.342588+0000","last_undegraded":"2024-04-29T20:48:28.342588+0000","last_fullsized":"2024-04-29T20:48:28.342588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.339050+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:54:13.801541+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1851,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.511450+0000","last_change":"2024-04-29T20:25:24.951540+0000","last_active":"2024-04-29T20:48:24.511450+0000","last_peered":"2024-04-29T20:48:24.511450+0000","last_clean":"2024-04-29T20:48:24.511450+0000","last_became_active":"2024-04-29T20:25:24.924922+0000","last_became_peered":"2024-04-29T20:25:24.924922+0000","last_unstale":"2024-04-29T20:48:24.511450+0000","last_undegraded":"2024-04-29T20:48:24.511450+0000","last_fullsized":"2024-04-29T20:48:24.511450+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1852,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.501659+0000","last_change":"2024-04-29T20:25:25.197139+0000","last_active":"2024-04-29T20:48:24.501659+0000","last_peered":"2024-04-29T20:48:24.501659+0000","last_clean":"2024-04-29T20:48:24.501659+0000","last_became_active":"2024-04-29T20:25:25.191769+0000","last_became_peered":"2024-04-29T20:25:25.191769+0000","last_unstale":"2024-04-29T20:48:24.501659+0000","last_undegraded":"2024-04-29T20:48:24.501659+0000","last_fullsized":"2024-04-29T20:48:24.501659+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1860,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.747835+0000","last_change":"2024-04-29T20:25:26.047119+0000","last_active":"2024-04-29T20:48:24.747835+0000","last_peered":"2024-04-29T20:48:24.747835+0000","last_clean":"2024-04-29T20:48:24.747835+0000","last_became_active":"2024-04-29T20:25:26.041502+0000","last_became_peered":"2024-04-29T20:25:26.041502+0000","last_unstale":"2024-04-29T20:48:24.747835+0000","last_undegraded":"2024-04-29T20:48:24.747835+0000","last_fullsized":"2024-04-29T20:48:24.747835+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:41:25.674969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1853,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.771787+0000","last_change":"2024-04-29T20:25:25.625395+0000","last_active":"2024-04-29T20:48:24.771787+0000","last_peered":"2024-04-29T20:48:24.771787+0000","last_clean":"2024-04-29T20:48:24.771787+0000","last_became_active":"2024-04-29T20:25:25.611019+0000","last_became_peered":"2024-04-29T20:25:25.611019+0000","last_unstale":"2024-04-29T20:48:24.771787+0000","last_undegraded":"2024-04-29T20:48:24.771787+0000","last_fullsized":"2024-04-29T20:48:24.771787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1871,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.766859+0000","last_change":"2024-04-29T20:25:22.691865+0000","last_active":"2024-04-29T20:48:24.766859+0000","last_peered":"2024-04-29T20:48:24.766859+0000","last_clean":"2024-04-29T20:48:24.766859+0000","last_became_active":"2024-04-29T20:25:22.685992+0000","last_became_peered":"2024-04-29T20:25:22.685992+0000","last_unstale":"2024-04-29T20:48:24.766859+0000","last_undegraded":"2024-04-29T20:48:24.766859+0000","last_fullsized":"2024-04-29T20:48:24.766859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:23:46.806032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1867,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.766205+0000","last_change":"2024-04-29T20:25:26.419912+0000","last_active":"2024-04-29T20:48:24.766205+0000","last_peered":"2024-04-29T20:48:24.766205+0000","last_clean":"2024-04-29T20:48:24.766205+0000","last_became_active":"2024-04-29T20:25:26.414222+0000","last_became_peered":"2024-04-29T20:25:26.414222+0000","last_unstale":"2024-04-29T20:48:24.766205+0000","last_undegraded":"2024-04-29T20:48:24.766205+0000","last_fullsized":"2024-04-29T20:48:24.766205+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1869,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.898365+0000","last_change":"2024-04-29T20:48:29.899776+0000","last_active":"2024-04-29T20:48:29.898365+0000","last_peered":"2024-04-29T20:48:29.898365+0000","last_clean":"2024-04-29T20:48:29.898365+0000","last_became_active":"2024-04-29T20:25:31.114951+0000","last_became_peered":"2024-04-29T20:25:31.114951+0000","last_unstale":"2024-04-29T20:48:29.898365+0000","last_undegraded":"2024-04-29T20:48:29.898365+0000","last_fullsized":"2024-04-29T20:48:29.898365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.895615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.895615+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.895615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:26:54.389350+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1864,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.500855+0000","last_change":"2024-04-29T20:25:27.559546+0000","last_active":"2024-04-29T20:48:24.500855+0000","last_peered":"2024-04-29T20:48:24.500855+0000","last_clean":"2024-04-29T20:48:24.500855+0000","last_became_active":"2024-04-29T20:25:27.553478+0000","last_became_peered":"2024-04-29T20:25:27.553478+0000","last_unstale":"2024-04-29T20:48:24.500855+0000","last_undegraded":"2024-04-29T20:48:24.500855+0000","last_fullsized":"2024-04-29T20:48:24.500855+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1873,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.763656+0000","last_change":"2024-04-29T20:25:06.859912+0000","last_active":"2024-04-29T20:48:24.763656+0000","last_peered":"2024-04-29T20:48:24.763656+0000","last_clean":"2024-04-29T20:48:24.763656+0000","last_became_active":"2024-04-29T20:25:06.850044+0000","last_became_peered":"2024-04-29T20:25:06.850044+0000","last_unstale":"2024-04-29T20:48:24.763656+0000","last_undegraded":"2024-04-29T20:48:24.763656+0000","last_fullsized":"2024-04-29T20:48:24.763656+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:27:20.439378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"976'8","reported_seq":1895,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:27.082294+0000","last_change":"2024-04-29T20:48:27.083951+0000","last_active":"2024-04-29T20:48:27.082294+0000","last_peered":"2024-04-29T20:48:27.082294+0000","last_clean":"2024-04-29T20:48:27.082294+0000","last_became_active":"2024-04-29T20:25:09.413652+0000","last_became_peered":"2024-04-29T20:25:09.413652+0000","last_unstale":"2024-04-29T20:48:27.082294+0000","last_undegraded":"2024-04-29T20:48:27.082294+0000","last_fullsized":"2024-04-29T20:48:27.082294+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'8","last_scrub_stamp":"2024-04-29T20:48:27.080520+0000","last_deep_scrub":"976'8","last_deep_scrub_stamp":"2024-04-29T20:48:27.080520+0000","last_clean_scrub_stamp":"2024-04-29T20:48:27.080520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:58:49.028677+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1865,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.751524+0000","last_change":"2024-04-29T20:25:21.119321+0000","last_active":"2024-04-29T20:48:24.751524+0000","last_peered":"2024-04-29T20:48:24.751524+0000","last_clean":"2024-04-29T20:48:24.751524+0000","last_became_active":"2024-04-29T20:25:21.109982+0000","last_became_peered":"2024-04-29T20:25:21.109982+0000","last_unstale":"2024-04-29T20:48:24.751524+0000","last_undegraded":"2024-04-29T20:48:24.751524+0000","last_fullsized":"2024-04-29T20:48:24.751524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:58:45.347986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.761695+0000","last_change":"2024-04-29T20:25:04.146851+0000","last_active":"2024-04-29T20:48:24.761695+0000","last_peered":"2024-04-29T20:48:24.761695+0000","last_clean":"2024-04-29T20:48:24.761695+0000","last_became_active":"2024-04-29T20:25:04.139931+0000","last_became_peered":"2024-04-29T20:25:04.139931+0000","last_unstale":"2024-04-29T20:48:24.761695+0000","last_undegraded":"2024-04-29T20:48:24.761695+0000","last_fullsized":"2024-04-29T20:48:24.761695+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:07:25.721846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"976'78","reported_seq":1899,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:27.039039+0000","last_change":"2024-04-29T20:48:27.040388+0000","last_active":"2024-04-29T20:48:27.039039+0000","last_peered":"2024-04-29T20:48:27.039039+0000","last_clean":"2024-04-29T20:48:27.039039+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:48:27.039039+0000","last_undegraded":"2024-04-29T20:48:27.039039+0000","last_fullsized":"2024-04-29T20:48:27.039039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'78","last_scrub_stamp":"2024-04-29T20:48:27.037292+0000","last_deep_scrub":"976'78","last_deep_scrub_stamp":"2024-04-29T20:48:27.037292+0000","last_clean_scrub_stamp":"2024-04-29T20:48:27.037292+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:42:36.706385+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1877,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.504231+0000","last_change":"2024-04-29T20:22:58.874338+0000","last_active":"2024-04-29T20:48:24.504231+0000","last_peered":"2024-04-29T20:48:24.504231+0000","last_clean":"2024-04-29T20:48:24.504231+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:48:24.504231+0000","last_undegraded":"2024-04-29T20:48:24.504231+0000","last_fullsized":"2024-04-29T20:48:24.504231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1863,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.499873+0000","last_change":"2024-04-29T20:25:07.978420+0000","last_active":"2024-04-29T20:48:24.499873+0000","last_peered":"2024-04-29T20:48:24.499873+0000","last_clean":"2024-04-29T20:48:24.499873+0000","last_became_active":"2024-04-29T20:25:07.961102+0000","last_became_peered":"2024-04-29T20:25:07.961102+0000","last_unstale":"2024-04-29T20:48:24.499873+0000","last_undegraded":"2024-04-29T20:48:24.499873+0000","last_fullsized":"2024-04-29T20:48:24.499873+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.693613+0000","last_change":"2024-04-29T20:48:28.695061+0000","last_active":"2024-04-29T20:48:28.693613+0000","last_peered":"2024-04-29T20:48:28.693613+0000","last_clean":"2024-04-29T20:48:28.693613+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:48:28.693613+0000","last_undegraded":"2024-04-29T20:48:28.693613+0000","last_fullsized":"2024-04-29T20:48:28.693613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.690925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.690925+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.690925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:42:53.149157+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1869,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.749714+0000","last_change":"2024-04-29T20:25:16.331754+0000","last_active":"2024-04-29T20:48:24.749714+0000","last_peered":"2024-04-29T20:48:24.749714+0000","last_clean":"2024-04-29T20:48:24.749714+0000","last_became_active":"2024-04-29T20:25:16.326218+0000","last_became_peered":"2024-04-29T20:25:16.326218+0000","last_unstale":"2024-04-29T20:48:24.749714+0000","last_undegraded":"2024-04-29T20:48:24.749714+0000","last_fullsized":"2024-04-29T20:48:24.749714+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:34:14.827307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1887,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.458783+0000","last_change":"2024-04-29T20:22:58.100523+0000","last_active":"2024-04-29T20:48:24.458783+0000","last_peered":"2024-04-29T20:48:24.458783+0000","last_clean":"2024-04-29T20:48:24.458783+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:48:24.458783+0000","last_undegraded":"2024-04-29T20:48:24.458783+0000","last_fullsized":"2024-04-29T20:48:24.458783+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:31:35.619213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1871,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.756311+0000","last_change":"2024-04-29T20:25:06.176161+0000","last_active":"2024-04-29T20:48:24.756311+0000","last_peered":"2024-04-29T20:48:24.756311+0000","last_clean":"2024-04-29T20:48:24.756311+0000","last_became_active":"2024-04-29T20:25:06.170624+0000","last_became_peered":"2024-04-29T20:25:06.170624+0000","last_unstale":"2024-04-29T20:48:24.756311+0000","last_undegraded":"2024-04-29T20:48:24.756311+0000","last_fullsized":"2024-04-29T20:48:24.756311+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:38:50.008456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1877,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.508343+0000","last_change":"2024-04-29T20:22:58.456344+0000","last_active":"2024-04-29T20:48:24.508343+0000","last_peered":"2024-04-29T20:48:24.508343+0000","last_clean":"2024-04-29T20:48:24.508343+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:48:24.508343+0000","last_undegraded":"2024-04-29T20:48:24.508343+0000","last_fullsized":"2024-04-29T20:48:24.508343+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1867,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.467301+0000","last_change":"2024-04-29T20:25:19.733222+0000","last_active":"2024-04-29T20:48:24.467301+0000","last_peered":"2024-04-29T20:48:24.467301+0000","last_clean":"2024-04-29T20:48:24.467301+0000","last_became_active":"2024-04-29T20:25:19.725027+0000","last_became_peered":"2024-04-29T20:25:19.725027+0000","last_unstale":"2024-04-29T20:48:24.467301+0000","last_undegraded":"2024-04-29T20:48:24.467301+0000","last_fullsized":"2024-04-29T20:48:24.467301+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:51:22.081409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1886,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.758437+0000","last_change":"2024-04-29T20:22:58.466585+0000","last_active":"2024-04-29T20:48:24.758437+0000","last_peered":"2024-04-29T20:48:24.758437+0000","last_clean":"2024-04-29T20:48:24.758437+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:48:24.758437+0000","last_undegraded":"2024-04-29T20:48:24.758437+0000","last_fullsized":"2024-04-29T20:48:24.758437+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:05:28.648611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1873,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.462478+0000","last_change":"2024-04-29T20:25:04.688244+0000","last_active":"2024-04-29T20:48:24.462478+0000","last_peered":"2024-04-29T20:48:24.462478+0000","last_clean":"2024-04-29T20:48:24.462478+0000","last_became_active":"2024-04-29T20:25:04.678852+0000","last_became_peered":"2024-04-29T20:25:04.678852+0000","last_unstale":"2024-04-29T20:48:24.462478+0000","last_undegraded":"2024-04-29T20:48:24.462478+0000","last_fullsized":"2024-04-29T20:48:24.462478+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:05:13.179776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.695724+0000","last_change":"2024-04-29T20:48:29.696557+0000","last_active":"2024-04-29T20:48:29.695724+0000","last_peered":"2024-04-29T20:48:29.695724+0000","last_clean":"2024-04-29T20:48:29.695724+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:48:29.695724+0000","last_undegraded":"2024-04-29T20:48:29.695724+0000","last_fullsized":"2024-04-29T20:48:29.695724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.694039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.694039+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.694039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:58:02.880900+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1879,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:26.015512+0000","last_change":"2024-04-29T20:48:26.016409+0000","last_active":"2024-04-29T20:48:26.015512+0000","last_peered":"2024-04-29T20:48:26.015512+0000","last_clean":"2024-04-29T20:48:26.015512+0000","last_became_active":"2024-04-29T20:25:05.045385+0000","last_became_peered":"2024-04-29T20:25:05.045385+0000","last_unstale":"2024-04-29T20:48:26.015512+0000","last_undegraded":"2024-04-29T20:48:26.015512+0000","last_fullsized":"2024-04-29T20:48:26.015512+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:26.010545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:26.010545+0000","last_clean_scrub_stamp":"2024-04-29T20:48:26.010545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:06:42.933639+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.180935+0000","last_change":"2024-04-29T20:48:29.185471+0000","last_active":"2024-04-29T20:48:29.180935+0000","last_peered":"2024-04-29T20:48:29.180935+0000","last_clean":"2024-04-29T20:48:29.180935+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:48:29.180935+0000","last_undegraded":"2024-04-29T20:48:29.180935+0000","last_fullsized":"2024-04-29T20:48:29.180935+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.176556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.176556+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.176556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:52:06.939200+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1879,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.951744+0000","last_change":"2024-04-29T20:48:28.952605+0000","last_active":"2024-04-29T20:48:28.951744+0000","last_peered":"2024-04-29T20:48:28.951744+0000","last_clean":"2024-04-29T20:48:28.951744+0000","last_became_active":"2024-04-29T20:25:19.206413+0000","last_became_peered":"2024-04-29T20:25:19.206413+0000","last_unstale":"2024-04-29T20:48:28.951744+0000","last_undegraded":"2024-04-29T20:48:28.951744+0000","last_fullsized":"2024-04-29T20:48:28.951744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.949663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.949663+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.949663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:18:18.247413+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1874,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.533729+0000","last_change":"2024-04-29T20:22:59.008821+0000","last_active":"2024-04-29T20:48:24.533729+0000","last_peered":"2024-04-29T20:48:24.533729+0000","last_clean":"2024-04-29T20:48:24.533729+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:48:24.533729+0000","last_undegraded":"2024-04-29T20:48:24.533729+0000","last_fullsized":"2024-04-29T20:48:24.533729+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:22:56.508104+0000","last_clean_scrub_stamp":"2024-04-29T20:22:56.508104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1868,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.272536+0000","last_change":"2024-04-29T20:48:29.273379+0000","last_active":"2024-04-29T20:48:29.272536+0000","last_peered":"2024-04-29T20:48:29.272536+0000","last_clean":"2024-04-29T20:48:29.272536+0000","last_became_active":"2024-04-29T20:25:19.589991+0000","last_became_peered":"2024-04-29T20:25:19.589991+0000","last_unstale":"2024-04-29T20:48:29.272536+0000","last_undegraded":"2024-04-29T20:48:29.272536+0000","last_fullsized":"2024-04-29T20:48:29.272536+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.270909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.270909+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.270909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:57:08.446947+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1853,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.759536+0000","last_change":"2024-04-29T20:25:24.677178+0000","last_active":"2024-04-29T20:48:24.759536+0000","last_peered":"2024-04-29T20:48:24.759536+0000","last_clean":"2024-04-29T20:48:24.759536+0000","last_became_active":"2024-04-29T20:25:24.662027+0000","last_became_peered":"2024-04-29T20:25:24.662027+0000","last_unstale":"2024-04-29T20:48:24.759536+0000","last_undegraded":"2024-04-29T20:48:24.759536+0000","last_fullsized":"2024-04-29T20:48:24.759536+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1863,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.531468+0000","last_change":"2024-04-29T20:25:26.584778+0000","last_active":"2024-04-29T20:48:24.531468+0000","last_peered":"2024-04-29T20:48:24.531468+0000","last_clean":"2024-04-29T20:48:24.531468+0000","last_became_active":"2024-04-29T20:25:26.576214+0000","last_became_peered":"2024-04-29T20:25:26.576214+0000","last_unstale":"2024-04-29T20:48:24.531468+0000","last_undegraded":"2024-04-29T20:48:24.531468+0000","last_fullsized":"2024-04-29T20:48:24.531468+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1873,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.475538+0000","last_change":"2024-04-29T20:25:02.740652+0000","last_active":"2024-04-29T20:48:24.475538+0000","last_peered":"2024-04-29T20:48:24.475538+0000","last_clean":"2024-04-29T20:48:24.475538+0000","last_became_active":"2024-04-29T20:25:02.734276+0000","last_became_peered":"2024-04-29T20:25:02.734276+0000","last_unstale":"2024-04-29T20:48:24.475538+0000","last_undegraded":"2024-04-29T20:48:24.475538+0000","last_fullsized":"2024-04-29T20:48:24.475538+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:34.054845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.752281+0000","last_change":"2024-04-29T20:25:06.026305+0000","last_active":"2024-04-29T20:48:24.752281+0000","last_peered":"2024-04-29T20:48:24.752281+0000","last_clean":"2024-04-29T20:48:24.752281+0000","last_became_active":"2024-04-29T20:25:06.011110+0000","last_became_peered":"2024-04-29T20:25:06.011110+0000","last_unstale":"2024-04-29T20:48:24.752281+0000","last_undegraded":"2024-04-29T20:48:24.752281+0000","last_fullsized":"2024-04-29T20:48:24.752281+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:51:47.148992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1874,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.456089+0000","last_change":"2024-04-29T20:25:02.267425+0000","last_active":"2024-04-29T20:48:24.456089+0000","last_peered":"2024-04-29T20:48:24.456089+0000","last_clean":"2024-04-29T20:48:24.456089+0000","last_became_active":"2024-04-29T20:25:02.256407+0000","last_became_peered":"2024-04-29T20:25:02.256407+0000","last_unstale":"2024-04-29T20:48:24.456089+0000","last_undegraded":"2024-04-29T20:48:24.456089+0000","last_fullsized":"2024-04-29T20:48:24.456089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:56:42.168617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:27.295944+0000","last_change":"2024-04-29T20:48:27.297036+0000","last_active":"2024-04-29T20:48:27.295944+0000","last_peered":"2024-04-29T20:48:27.295944+0000","last_clean":"2024-04-29T20:48:27.295944+0000","last_became_active":"2024-04-29T20:25:05.517205+0000","last_became_peered":"2024-04-29T20:25:05.517205+0000","last_unstale":"2024-04-29T20:48:27.295944+0000","last_undegraded":"2024-04-29T20:48:27.295944+0000","last_fullsized":"2024-04-29T20:48:27.295944+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:27.294294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:27.294294+0000","last_clean_scrub_stamp":"2024-04-29T20:48:27.294294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:00:01.942619+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1875,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.034568+0000","last_change":"2024-04-29T20:48:28.034568+0000","last_active":"2024-04-29T20:48:28.034568+0000","last_peered":"2024-04-29T20:48:28.034568+0000","last_clean":"2024-04-29T20:48:28.034568+0000","last_became_active":"2024-04-29T20:25:11.174336+0000","last_became_peered":"2024-04-29T20:25:11.174336+0000","last_unstale":"2024-04-29T20:48:28.034568+0000","last_undegraded":"2024-04-29T20:48:28.034568+0000","last_fullsized":"2024-04-29T20:48:28.034568+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.028636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.028636+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.028636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:53:55.590130+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1863,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:25.250255+0000","last_change":"2024-04-29T20:25:09.447149+0000","last_active":"2024-04-29T20:48:25.250255+0000","last_peered":"2024-04-29T20:48:25.250255+0000","last_clean":"2024-04-29T20:48:25.250255+0000","last_became_active":"2024-04-29T20:25:09.429696+0000","last_became_peered":"2024-04-29T20:25:09.429696+0000","last_unstale":"2024-04-29T20:48:25.250255+0000","last_undegraded":"2024-04-29T20:48:25.250255+0000","last_fullsized":"2024-04-29T20:48:25.250255+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1870,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:24.769020+0000","last_change":"2024-04-29T20:25:22.557015+0000","last_active":"2024-04-29T20:48:24.769020+0000","last_peered":"2024-04-29T20:48:24.769020+0000","last_clean":"2024-04-29T20:48:24.769020+0000","last_became_active":"2024-04-29T20:25:22.551405+0000","last_became_peered":"2024-04-29T20:25:22.551405+0000","last_unstale":"2024-04-29T20:48:24.769020+0000","last_undegraded":"2024-04-29T20:48:24.769020+0000","last_fullsized":"2024-04-29T20:48:24.769020+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:24:52.586982+0000","last_clean_scrub_stamp":"2024-04-29T20:24:52.586982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:48:10.829532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.875567+0000","last_change":"2024-04-29T20:48:28.876595+0000","last_active":"2024-04-29T20:48:28.875567+0000","last_peered":"2024-04-29T20:48:28.875567+0000","last_clean":"2024-04-29T20:48:28.875567+0000","last_became_active":"2024-04-29T20:25:12.263826+0000","last_became_peered":"2024-04-29T20:25:12.263826+0000","last_unstale":"2024-04-29T20:48:28.875567+0000","last_undegraded":"2024-04-29T20:48:28.875567+0000","last_fullsized":"2024-04-29T20:48:28.875567+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.871622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.871622+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.871622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:19:26.221756+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1878,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.030280+0000","last_change":"2024-04-29T20:48:28.031631+0000","last_active":"2024-04-29T20:48:28.030280+0000","last_peered":"2024-04-29T20:48:28.030280+0000","last_clean":"2024-04-29T20:48:28.030280+0000","last_became_active":"2024-04-29T20:25:09.849524+0000","last_became_peered":"2024-04-29T20:25:09.849524+0000","last_unstale":"2024-04-29T20:48:28.030280+0000","last_undegraded":"2024-04-29T20:48:28.030280+0000","last_fullsized":"2024-04-29T20:48:28.030280+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.025453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.025453+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.025453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:14:13.315429+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280823,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81068,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290772,"statfs":{"total":96636764160,"available":96553750528,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280823,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81464,"kb_used_data":5872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290376,"statfs":{"total":96636764160,"available":96553345024,"internally_reserved":0,"allocated":6012928,"data_stored":3244619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280822,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68592,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303248,"statfs":{"total":96636764160,"available":96566525952,"internally_reserved":0,"allocated":5611520,"data_stored":2846779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":6,"up_from":20,"seq":85899346233,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81072,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290768,"statfs":{"total":96636764160,"available":96553746432,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81072,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290768,"statfs":{"total":96636764160,"available":96553746432,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509751,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68980,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302860,"statfs":{"total":96636764160,"available":96566128640,"internally_reserved":0,"allocated":6008832,"data_stored":3244614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509752,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509749,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81068,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290772,"statfs":{"total":96636764160,"available":96553750528,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.1a last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.19 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.18 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.7 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.6 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.5 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.598 INFO:tasks.ceph:pgid 11.3 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 11.2 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 11.0 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 11.a last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-29T20:22:56.508104+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=22, tm_sec=56, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 11.9 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-29T20:22:56.508104+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=22, tm_sec=56, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 11.b last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.599 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-29T20:22:56.508104+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=22, tm_sec=56, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 11.d last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-29T20:22:56.508104+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=22, tm_sec=56, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 11.c last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-29T20:22:56.508104+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=22, tm_sec=56, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 11.1d last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 11.1e last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 11.1f last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.600 INFO:tasks.ceph:pgid 11.10 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.601 INFO:tasks.ceph:pgid 11.11 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.601 INFO:tasks.ceph:pgid 11.14 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.601 INFO:tasks.ceph:pgid 11.15 last_scrub_stamp 2024-04-29T20:24:52.586982+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=24, tm_sec=52, tm_wday=0, tm_yday=120, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=29, tm_hour=20, tm_min=48, tm_sec=23, tm_wday=0, tm_yday=120, tm_isdst=0) 2024-04-29T20:48:32.601 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-29T20:48:34.683 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:53.190 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.693 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:53.200 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.711 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:26:53.218 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.733 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:53.211 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.736 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:53.243 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.738 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:53.217 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.742 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:53.249 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.757 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:26:53.235 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:34.760 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:26:53.267 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.596 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:08.103 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.598 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:08.105 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.613 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:08.121 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.651 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:08.129 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.652 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:08.130 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.654 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:08.161 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.655 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:08.163 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.668 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:08.147 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:49.727 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:08.234 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.366 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:08.844 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.367 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:08.846 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.375 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:08.853 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.395 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:08.902 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.396 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:08.903 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.407 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:08.914 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.448 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:08.956 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.449 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:08.957 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:50.459 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:08.967 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:48:52.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.712+0000 7f8b52196640 10 monclient: get_monmap_and_config 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.712+0000 7f8b52196640 10 monclient: build_initial_monmap 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.712+0000 7f8b52196640 10 monclient: monmap: 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:52.713111+0000 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:52.713111+0000 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:52.712 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:52.713 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:52.713 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:52.713 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:52.713 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:52.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 1 Processor -- start 2024-04-29T20:48:52.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 1 -- start start 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 10 monclient: init 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b4c0b9570 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b4c0b3240 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c140e30 0x7f8b4c1471a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 10 monclient(hunting): picked mon.noname-b con 0x7f8b4c140e30 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b4c0abd80 con 0x7f8b4c0b3240 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.713+0000 7f8b52196640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b4c0aac70 con 0x7f8b4c0b9570 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b52196640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b52196640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4c0b9950 con 0x7f8b4c140e30 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b52196640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b52196640 10 monclient(hunting): authenticate will time out at 3081.873578s 2024-04-29T20:48:52.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b4bfff640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c140e30 0x7f8b4c1471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b42ffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8b4c0b3240 0x7f8b4c146a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59716/0 (socket says 172.21.15.64:59716) 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b42ffd640 1 -- 172.21.15.64:0/4000920369 learned_addr learned my addr 172.21.15.64:0/4000920369 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b4bfff640 10 monclient(hunting): get_auth_request con 0x7f8b4c140e30 auth_method 0 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b4bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.714+0000 7f8b4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:52.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.715+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.715+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.715+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_done global_id 16418 payload 274 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 1 -- 172.21.15.64:0/4000920369 >> v1:172.21.15.167:6789/0 conn(0x7f8b4c0b9570 legacy=0x7f8b4c140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 1 -- 172.21.15.64:0/4000920369 >> v1:172.21.15.64:6789/0 conn(0x7f8b4c0b3240 legacy=0x7f8b4c146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 10 monclient: _finish_hunting 0 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 1 monclient: found mon.noname-b 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.64:3301/0 2024-04-29T20:48:52.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.717+0000 7f8b4bfff640 1 -- 172.21.15.64:0/4000920369 --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c0abff0 con 0x7f8b4c140e30 2024-04-29T20:48:52.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.719+0000 7f8b4bfff640 1 --2- 172.21.15.64:0/4000920369 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c140e30 0x7f8b4c1471a0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009dc0 tx=0x7f8b4402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=109c878d669cea18 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:52.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 1 -- 172.21.15.64:0/4000920369 <== mon.2 v2:172.21.15.64:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b44039070 con 0x7f8b4c140e30 2024-04-29T20:48:52.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:52.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:52.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: dump: 2024-04-29T20:48:52.722 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:52.722 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: _finish_auth 0 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b52196640 5 monclient: authenticate success, global_id 16418 2024-04-29T20:48:52.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b52196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 1 -- 172.21.15.64:0/4000920369 <== mon.2 v2:172.21.15.64:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b4402cd50 con 0x7f8b4c140e30 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b52196640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b52196640 4 set_mon_vals no callback set 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 1 -- 172.21.15.64:0/4000920369 <== mon.2 v2:172.21.15.64:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b44034660 con 0x7f8b4c140e30 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.722+0000 7f8b4affd640 10 monclient: dump: 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:52.727 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 10 monclient: shutdown 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 -- 172.21.15.64:0/4000920369 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c140e30 msgr2=0x7f8b4c1471a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 --2- 172.21.15.64:0/4000920369 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c140e30 0x7f8b4c1471a0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009dc0 tx=0x7f8b4402bfa0 comp rx=0 tx=0).stop 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 -- 172.21.15.64:0/4000920369 shutdown_connections 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 --2- 172.21.15.64:0/4000920369 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c140e30 0x7f8b4c1471a0 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 -- 172.21.15.64:0/4000920369 >> 172.21.15.64:0/4000920369 conn(0x7f8b4c06cba0 msgr2=0x7f8b4c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 -- 172.21.15.64:0/4000920369 shutdown_connections 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.723+0000 7f8b52196640 1 -- 172.21.15.64:0/4000920369 wait complete. 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.724+0000 7f8b52196640 10 monclient: build_initial_monmap 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.724+0000 7f8b52196640 10 monclient: monmap: 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:52.729 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.724+0000 7f8b52196640 1 librados: starting msgr at 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.724+0000 7f8b52196640 1 librados: starting objecter 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.724+0000 7f8b52196640 1 Processor -- start 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 -- start start 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 librados: setting wanted keys 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 librados: calling monclient init 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient: init 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-29T20:48:52.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c0b3240 0x7f8b4c143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): picked mon.noname-c con 0x7f8b4c0b3240 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b4c0b9570 0x7f8b4c0ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): picked mon.noname-b con 0x7f8b4c0b9570 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b4c140e30 0x7f8b4c13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): picked mon.noname-a con 0x7f8b4c140e30 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c14edf0 con 0x7f8b4c140e30 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c1476e0 con 0x7f8b4c0b9570 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4c14ef70 con 0x7f8b4c0b3240 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.725+0000 7f8b52196640 10 monclient(hunting): authenticate will time out at 3081.885122s 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b42ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b4c0b9570 0x7f8b4c0ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:52.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b4c140e30 0x7f8b4c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c0b3240 0x7f8b4c143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c0b3240 0x7f8b4c143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3301/0 says I am v2:172.21.15.64:55752/0 (socket says 172.21.15.64:55752) 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 1 -- 172.21.15.64:0/3942927704 learned_addr learned my addr 172.21.15.64:0/3942927704 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 10 monclient(hunting): get_auth_request con 0x7f8b4c0b3240 auth_method 0 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b42ffd640 10 monclient(hunting): get_auth_request con 0x7f8b4c0b9570 auth_method 0 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b42ffd640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4bfff640 10 monclient(hunting): get_auth_request con 0x7f8b4c140e30 auth_method 0 2024-04-29T20:48:52.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4bfff640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.726+0000 7f8b4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:52.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.727+0000 7f8b4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:52.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 10 monclient(hunting): handle_auth_done global_id 22222 payload 824 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 1 -- 172.21.15.64:0/3942927704 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c0b3240 msgr2=0x7f8b4c143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c0b3240 0x7f8b4c143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 1 -- 172.21.15.64:0/3942927704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b4c140e30 msgr2=0x7f8b4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b4c140e30 0x7f8b4c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 10 monclient: _finish_hunting 0 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 1 monclient: found mon.noname-b 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.729+0000 7f8b42ffd640 1 -- 172.21.15.64:0/3942927704 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c0ad280 con 0x7f8b4c0b9570 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.732+0000 7f8b42ffd640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b4c0b9570 0x7f8b4c0ad980 secure :-1 s=READY pgs=5648 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c440 tx=0x7f8b34001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:52.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.732+0000 7f8b48ff9640 10 client.?.objecter ms_handle_connect 0x7f8b4c0b9570 2024-04-29T20:48:52.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.732+0000 7f8b48ff9640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34017070 con 0x7f8b4c0b9570 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: dump: 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:52.737 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: _finish_auth 0 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: _check_auth_tickets 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b34005ce0 con 0x7f8b4c0b9570 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b340072d0 con 0x7f8b4c0b9570 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 5 monclient: authenticate success, global_id 22222 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 10 monclient: _renew_subs 2024-04-29T20:48:52.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4c0ad560 con 0x7f8b4c0b9570 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b4a7fc640 4 set_mon_vals no callback set 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b48ff9640 10 monclient: dump: 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:52.739 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 10 client.22222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 10 monclient: _renew_subs 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b4c13d0a0 con 0x7f8b4c0b9570 2024-04-29T20:48:52.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.736+0000 7f8b52196640 1 librados: init done 2024-04-29T20:48:52.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.737+0000 7f8b52196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:52.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.737+0000 7f8b52196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:52.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.737+0000 7f8b52196640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:52.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.737+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4c04fce0 con 0x7f8b4c0b9570 2024-04-29T20:48:52.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.755+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b3401a070 con 0x7f8b4c0b9570 2024-04-29T20:48:52.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.756+0000 7f8b48ff9640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f8b2c06e280 0x7f8b2c070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:52.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.756+0000 7f8b4b7fe640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f8b2c06e280 0x7f8b2c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:52.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.756+0000 7f8b4b7fe640 10 monclient: get_auth_request con 0x7f8b2c06e280 auth_method 0 2024-04-29T20:48:52.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.757+0000 7f8b48ff9640 10 client.22222.objecter ms_handle_connect 0x7f8b2c06e280 2024-04-29T20:48:52.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.757+0000 7f8b4b7fe640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f8b2c06e280 0x7f8b2c070740 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c007fd0 tx=0x7f8b3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.758+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f8b340571c0 con 0x7f8b4c0b9570 2024-04-29T20:48:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.758+0000 7f8b48ff9640 10 client.22222.objecter ms_dispatch 0x7f8b4c000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.758+0000 7f8b48ff9640 3 client.22222.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.758+0000 7f8b48ff9640 3 client.22222.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:52.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:52.758+0000 7f8b48ff9640 20 client.22222.objecter dump_active .. 0 homeless 2024-04-29T20:48:53.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.000+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b34059120 con 0x7f8b4c0b9570 2024-04-29T20:48:53.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.000+0000 7f8b48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:53.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.000+0000 7f8b48ff9640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:53.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.200+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 --> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b4c13c7a0 con 0x7f8b2c06e280 2024-04-29T20:48:53.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.204+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3942927704 <== mgr.4108 v2:172.21.15.167:6800/3900323440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116157 (secure 0 0 0) 0x7f8b4c13c7a0 con 0x7f8b2c06e280 2024-04-29T20:48:53.205 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-29T20:48:53.205 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-29T20:48:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 10 librados: watch_flush enter 2024-04-29T20:48:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 10 librados: watch_flush exit 2024-04-29T20:48:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 20 client.22222.objecter shutdown clearing up homeless session... 2024-04-29T20:48:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 10 client.22222.objecter successfully canceled tick 2024-04-29T20:48:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f8b2c06e280 msgr2=0x7f8b2c070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f8b2c06e280 0x7f8b2c070740 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c007fd0 tx=0x7f8b3c009040 comp rx=0 tx=0).stop 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 10 monclient: shutdown 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b4c0b9570 msgr2=0x7f8b4c0ad980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b4c0b9570 0x7f8b4c0ad980 secure :-1 s=READY pgs=5648 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c440 tx=0x7f8b34001e70 comp rx=0 tx=0).stop 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 shutdown_connections 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f8b2c06e280 0x7f8b2c070740 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b4c140e30 0x7f8b4c13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b4c0b9570 0x7f8b4c0ad980 unknown :-1 s=CLOSED pgs=5648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 --2- 172.21.15.64:0/3942927704 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f8b4c0b3240 0x7f8b4c143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 >> 172.21.15.64:0/3942927704 conn(0x7f8b4c06cba0 msgr2=0x7f8b4c0b7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 shutdown_connections 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 -- 172.21.15.64:0/3942927704 wait complete. 2024-04-29T20:48:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.206+0000 7f8b52196640 1 librados: shutdown 2024-04-29T20:48:53.217 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2024-04-29T20:48:51.637024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":616056,"kb_used_data":44600,"kb_used_omap":12,"kb_used_meta":571379,"kb_avail":754358664,"statfs":{"total":773094113280,"available":772463271936,"internally_reserved":0,"allocated":45670400,"data_stored":23569882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":585092658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002931"},"pg_stats":[{"pgid":"11.1b","version":"0'0","reported_seq":1869,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.342588+0000","last_change":"2024-04-29T20:48:28.343429+0000","last_active":"2024-04-29T20:48:28.342588+0000","last_peered":"2024-04-29T20:48:28.342588+0000","last_clean":"2024-04-29T20:48:28.342588+0000","last_became_active":"2024-04-29T20:25:19.211321+0000","last_became_peered":"2024-04-29T20:25:19.211321+0000","last_unstale":"2024-04-29T20:48:28.342588+0000","last_undegraded":"2024-04-29T20:48:28.342588+0000","last_fullsized":"2024-04-29T20:48:28.342588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.339050+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:54:13.801541+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1859,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:34.638763+0000","last_change":"2024-04-29T20:48:34.639588+0000","last_active":"2024-04-29T20:48:34.638763+0000","last_peered":"2024-04-29T20:48:34.638763+0000","last_clean":"2024-04-29T20:48:34.638763+0000","last_became_active":"2024-04-29T20:25:24.924922+0000","last_became_peered":"2024-04-29T20:25:24.924922+0000","last_unstale":"2024-04-29T20:48:34.638763+0000","last_undegraded":"2024-04-29T20:48:34.638763+0000","last_fullsized":"2024-04-29T20:48:34.638763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:34.637127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:34.637127+0000","last_clean_scrub_stamp":"2024-04-29T20:48:34.637127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:33:15.361854+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1860,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:30.987864+0000","last_change":"2024-04-29T20:48:30.988649+0000","last_active":"2024-04-29T20:48:30.987864+0000","last_peered":"2024-04-29T20:48:30.987864+0000","last_clean":"2024-04-29T20:48:30.987864+0000","last_became_active":"2024-04-29T20:25:25.191769+0000","last_became_peered":"2024-04-29T20:25:25.191769+0000","last_unstale":"2024-04-29T20:48:30.987864+0000","last_undegraded":"2024-04-29T20:48:30.987864+0000","last_fullsized":"2024-04-29T20:48:30.987864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:30.986200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:30.986200+0000","last_clean_scrub_stamp":"2024-04-29T20:48:30.986200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:06:20.908576+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1869,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:34.446110+0000","last_change":"2024-04-29T20:48:34.446902+0000","last_active":"2024-04-29T20:48:34.446110+0000","last_peered":"2024-04-29T20:48:34.446110+0000","last_clean":"2024-04-29T20:48:34.446110+0000","last_became_active":"2024-04-29T20:25:26.041502+0000","last_became_peered":"2024-04-29T20:25:26.041502+0000","last_unstale":"2024-04-29T20:48:34.446110+0000","last_undegraded":"2024-04-29T20:48:34.446110+0000","last_fullsized":"2024-04-29T20:48:34.446110+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:34.444477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:34.444477+0000","last_clean_scrub_stamp":"2024-04-29T20:48:34.444477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:40:41.892624+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1862,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:30.992962+0000","last_change":"2024-04-29T20:48:30.992962+0000","last_active":"2024-04-29T20:48:30.992962+0000","last_peered":"2024-04-29T20:48:30.992962+0000","last_clean":"2024-04-29T20:48:30.992962+0000","last_became_active":"2024-04-29T20:25:25.611019+0000","last_became_peered":"2024-04-29T20:25:25.611019+0000","last_unstale":"2024-04-29T20:48:30.992962+0000","last_undegraded":"2024-04-29T20:48:30.992962+0000","last_fullsized":"2024-04-29T20:48:30.992962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:30.986809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:30.986809+0000","last_clean_scrub_stamp":"2024-04-29T20:48:30.986809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:24:59.184087+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1881,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:35.043461+0000","last_change":"2024-04-29T20:48:35.043461+0000","last_active":"2024-04-29T20:48:35.043461+0000","last_peered":"2024-04-29T20:48:35.043461+0000","last_clean":"2024-04-29T20:48:35.043461+0000","last_became_active":"2024-04-29T20:25:22.685992+0000","last_became_peered":"2024-04-29T20:25:22.685992+0000","last_unstale":"2024-04-29T20:48:35.043461+0000","last_undegraded":"2024-04-29T20:48:35.043461+0000","last_fullsized":"2024-04-29T20:48:35.043461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:35.041042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:35.041042+0000","last_clean_scrub_stamp":"2024-04-29T20:48:35.041042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:17:44.615277+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1875,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:31.973097+0000","last_change":"2024-04-29T20:48:31.974539+0000","last_active":"2024-04-29T20:48:31.973097+0000","last_peered":"2024-04-29T20:48:31.973097+0000","last_clean":"2024-04-29T20:48:31.973097+0000","last_became_active":"2024-04-29T20:25:26.414222+0000","last_became_peered":"2024-04-29T20:25:26.414222+0000","last_unstale":"2024-04-29T20:48:31.973097+0000","last_undegraded":"2024-04-29T20:48:31.973097+0000","last_fullsized":"2024-04-29T20:48:31.973097+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:31.970372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:31.970372+0000","last_clean_scrub_stamp":"2024-04-29T20:48:31.970372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T20:51:24.947466+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1869,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.898365+0000","last_change":"2024-04-29T20:48:29.899776+0000","last_active":"2024-04-29T20:48:29.898365+0000","last_peered":"2024-04-29T20:48:29.898365+0000","last_clean":"2024-04-29T20:48:29.898365+0000","last_became_active":"2024-04-29T20:25:31.114951+0000","last_became_peered":"2024-04-29T20:25:31.114951+0000","last_unstale":"2024-04-29T20:48:29.898365+0000","last_undegraded":"2024-04-29T20:48:29.898365+0000","last_fullsized":"2024-04-29T20:48:29.898365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.895615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.895615+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.895615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:26:54.389350+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:36.697350+0000","last_change":"2024-04-29T20:48:36.698179+0000","last_active":"2024-04-29T20:48:36.697350+0000","last_peered":"2024-04-29T20:48:36.697350+0000","last_clean":"2024-04-29T20:48:36.697350+0000","last_became_active":"2024-04-29T20:25:27.553478+0000","last_became_peered":"2024-04-29T20:25:27.553478+0000","last_unstale":"2024-04-29T20:48:36.697350+0000","last_undegraded":"2024-04-29T20:48:36.697350+0000","last_fullsized":"2024-04-29T20:48:36.697350+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:36.695676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:36.695676+0000","last_clean_scrub_stamp":"2024-04-29T20:48:36.695676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:46:11.897813+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1882,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:32.941442+0000","last_change":"2024-04-29T20:48:32.942305+0000","last_active":"2024-04-29T20:48:32.941442+0000","last_peered":"2024-04-29T20:48:32.941442+0000","last_clean":"2024-04-29T20:48:32.941442+0000","last_became_active":"2024-04-29T20:25:06.850044+0000","last_became_peered":"2024-04-29T20:25:06.850044+0000","last_unstale":"2024-04-29T20:48:32.941442+0000","last_undegraded":"2024-04-29T20:48:32.941442+0000","last_fullsized":"2024-04-29T20:48:32.941442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:32.939772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:32.939772+0000","last_clean_scrub_stamp":"2024-04-29T20:48:32.939772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:45:33.909284+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"976'8","reported_seq":1895,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:27.082294+0000","last_change":"2024-04-29T20:48:27.083951+0000","last_active":"2024-04-29T20:48:27.082294+0000","last_peered":"2024-04-29T20:48:27.082294+0000","last_clean":"2024-04-29T20:48:27.082294+0000","last_became_active":"2024-04-29T20:25:09.413652+0000","last_became_peered":"2024-04-29T20:25:09.413652+0000","last_unstale":"2024-04-29T20:48:27.082294+0000","last_undegraded":"2024-04-29T20:48:27.082294+0000","last_fullsized":"2024-04-29T20:48:27.082294+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'8","last_scrub_stamp":"2024-04-29T20:48:27.080520+0000","last_deep_scrub":"976'8","last_deep_scrub_stamp":"2024-04-29T20:48:27.080520+0000","last_clean_scrub_stamp":"2024-04-29T20:48:27.080520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:58:49.028677+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1874,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:33.481219+0000","last_change":"2024-04-29T20:48:33.482040+0000","last_active":"2024-04-29T20:48:33.481219+0000","last_peered":"2024-04-29T20:48:33.481219+0000","last_clean":"2024-04-29T20:48:33.481219+0000","last_became_active":"2024-04-29T20:25:21.109982+0000","last_became_peered":"2024-04-29T20:25:21.109982+0000","last_unstale":"2024-04-29T20:48:33.481219+0000","last_undegraded":"2024-04-29T20:48:33.481219+0000","last_fullsized":"2024-04-29T20:48:33.481219+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:33.475845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:33.475845+0000","last_clean_scrub_stamp":"2024-04-29T20:48:33.475845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:01:16.336635+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1881,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:30.906317+0000","last_change":"2024-04-29T20:48:30.907137+0000","last_active":"2024-04-29T20:48:30.906317+0000","last_peered":"2024-04-29T20:48:30.906317+0000","last_clean":"2024-04-29T20:48:30.906317+0000","last_became_active":"2024-04-29T20:25:04.139931+0000","last_became_peered":"2024-04-29T20:25:04.139931+0000","last_unstale":"2024-04-29T20:48:30.906317+0000","last_undegraded":"2024-04-29T20:48:30.906317+0000","last_fullsized":"2024-04-29T20:48:30.906317+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:30.904591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:30.904591+0000","last_clean_scrub_stamp":"2024-04-29T20:48:30.904591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:12:08.687844+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"976'78","reported_seq":1899,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:27.039039+0000","last_change":"2024-04-29T20:48:27.040388+0000","last_active":"2024-04-29T20:48:27.039039+0000","last_peered":"2024-04-29T20:48:27.039039+0000","last_clean":"2024-04-29T20:48:27.039039+0000","last_became_active":"2024-04-29T20:22:57.023804+0000","last_became_peered":"2024-04-29T20:22:57.023804+0000","last_unstale":"2024-04-29T20:48:27.039039+0000","last_undegraded":"2024-04-29T20:48:27.039039+0000","last_fullsized":"2024-04-29T20:48:27.039039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'78","last_scrub_stamp":"2024-04-29T20:48:27.037292+0000","last_deep_scrub":"976'78","last_deep_scrub_stamp":"2024-04-29T20:48:27.037292+0000","last_clean_scrub_stamp":"2024-04-29T20:48:27.037292+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:42:36.706385+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:31.675866+0000","last_change":"2024-04-29T20:48:31.676797+0000","last_active":"2024-04-29T20:48:31.675866+0000","last_peered":"2024-04-29T20:48:31.675866+0000","last_clean":"2024-04-29T20:48:31.675866+0000","last_became_active":"2024-04-29T20:22:58.866574+0000","last_became_peered":"2024-04-29T20:22:58.866574+0000","last_unstale":"2024-04-29T20:48:31.675866+0000","last_undegraded":"2024-04-29T20:48:31.675866+0000","last_fullsized":"2024-04-29T20:48:31.675866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:31.674083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:31.674083+0000","last_clean_scrub_stamp":"2024-04-29T20:48:31.674083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:11:41.615941+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1871,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.992216+0000","last_change":"2024-04-29T20:48:29.993300+0000","last_active":"2024-04-29T20:48:29.992216+0000","last_peered":"2024-04-29T20:48:29.992216+0000","last_clean":"2024-04-29T20:48:29.992216+0000","last_became_active":"2024-04-29T20:25:07.961102+0000","last_became_peered":"2024-04-29T20:25:07.961102+0000","last_unstale":"2024-04-29T20:48:29.992216+0000","last_undegraded":"2024-04-29T20:48:29.992216+0000","last_fullsized":"2024-04-29T20:48:29.992216+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.990552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.990552+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.990552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:05:20.746627+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.693613+0000","last_change":"2024-04-29T20:48:28.695061+0000","last_active":"2024-04-29T20:48:28.693613+0000","last_peered":"2024-04-29T20:48:28.693613+0000","last_clean":"2024-04-29T20:48:28.693613+0000","last_became_active":"2024-04-29T20:22:58.128156+0000","last_became_peered":"2024-04-29T20:22:58.128156+0000","last_unstale":"2024-04-29T20:48:28.693613+0000","last_undegraded":"2024-04-29T20:48:28.693613+0000","last_fullsized":"2024-04-29T20:48:28.693613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.690925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.690925+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.690925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:42:53.149157+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1878,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:32.467158+0000","last_change":"2024-04-29T20:48:32.468028+0000","last_active":"2024-04-29T20:48:32.467158+0000","last_peered":"2024-04-29T20:48:32.467158+0000","last_clean":"2024-04-29T20:48:32.467158+0000","last_became_active":"2024-04-29T20:25:16.326218+0000","last_became_peered":"2024-04-29T20:25:16.326218+0000","last_unstale":"2024-04-29T20:48:32.467158+0000","last_undegraded":"2024-04-29T20:48:32.467158+0000","last_fullsized":"2024-04-29T20:48:32.467158+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:32.465495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:32.465495+0000","last_clean_scrub_stamp":"2024-04-29T20:48:32.465495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T01:04:00.127204+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1896,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:31.948607+0000","last_change":"2024-04-29T20:48:31.949488+0000","last_active":"2024-04-29T20:48:31.948607+0000","last_peered":"2024-04-29T20:48:31.948607+0000","last_clean":"2024-04-29T20:48:31.948607+0000","last_became_active":"2024-04-29T20:22:58.071426+0000","last_became_peered":"2024-04-29T20:22:58.071426+0000","last_unstale":"2024-04-29T20:48:31.948607+0000","last_undegraded":"2024-04-29T20:48:31.948607+0000","last_fullsized":"2024-04-29T20:48:31.948607+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:31.946968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:31.946968+0000","last_clean_scrub_stamp":"2024-04-29T20:48:31.946968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:06:15.394229+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1880,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:31.468175+0000","last_change":"2024-04-29T20:48:31.469218+0000","last_active":"2024-04-29T20:48:31.468175+0000","last_peered":"2024-04-29T20:48:31.468175+0000","last_clean":"2024-04-29T20:48:31.468175+0000","last_became_active":"2024-04-29T20:25:06.170624+0000","last_became_peered":"2024-04-29T20:25:06.170624+0000","last_unstale":"2024-04-29T20:48:31.468175+0000","last_undegraded":"2024-04-29T20:48:31.468175+0000","last_fullsized":"2024-04-29T20:48:31.468175+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:31.466531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:31.466531+0000","last_clean_scrub_stamp":"2024-04-29T20:48:31.466531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:16:34.983245+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:30.688710+0000","last_change":"2024-04-29T20:48:30.689564+0000","last_active":"2024-04-29T20:48:30.688710+0000","last_peered":"2024-04-29T20:48:30.688710+0000","last_clean":"2024-04-29T20:48:30.688710+0000","last_became_active":"2024-04-29T20:22:58.450795+0000","last_became_peered":"2024-04-29T20:22:58.450795+0000","last_unstale":"2024-04-29T20:48:30.688710+0000","last_undegraded":"2024-04-29T20:48:30.688710+0000","last_fullsized":"2024-04-29T20:48:30.688710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:30.684692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:30.684692+0000","last_clean_scrub_stamp":"2024-04-29T20:48:30.684692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:53:00.938583+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1876,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:36.063106+0000","last_change":"2024-04-29T20:48:36.063870+0000","last_active":"2024-04-29T20:48:36.063106+0000","last_peered":"2024-04-29T20:48:36.063106+0000","last_clean":"2024-04-29T20:48:36.063106+0000","last_became_active":"2024-04-29T20:25:19.725027+0000","last_became_peered":"2024-04-29T20:25:19.725027+0000","last_unstale":"2024-04-29T20:48:36.063106+0000","last_undegraded":"2024-04-29T20:48:36.063106+0000","last_fullsized":"2024-04-29T20:48:36.063106+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:36.061473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:36.061473+0000","last_clean_scrub_stamp":"2024-04-29T20:48:36.061473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:26:57.339604+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1895,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.893589+0000","last_change":"2024-04-29T20:48:29.894435+0000","last_active":"2024-04-29T20:48:29.893589+0000","last_peered":"2024-04-29T20:48:29.893589+0000","last_clean":"2024-04-29T20:48:29.893589+0000","last_became_active":"2024-04-29T20:22:58.459375+0000","last_became_peered":"2024-04-29T20:22:58.459375+0000","last_unstale":"2024-04-29T20:48:29.893589+0000","last_undegraded":"2024-04-29T20:48:29.893589+0000","last_fullsized":"2024-04-29T20:48:29.893589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.891916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.891916+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.891916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:28:25.368570+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1882,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:35.027213+0000","last_change":"2024-04-29T20:48:35.027993+0000","last_active":"2024-04-29T20:48:35.027213+0000","last_peered":"2024-04-29T20:48:35.027213+0000","last_clean":"2024-04-29T20:48:35.027213+0000","last_became_active":"2024-04-29T20:25:04.678852+0000","last_became_peered":"2024-04-29T20:25:04.678852+0000","last_unstale":"2024-04-29T20:48:35.027213+0000","last_undegraded":"2024-04-29T20:48:35.027213+0000","last_fullsized":"2024-04-29T20:48:35.027213+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:35.025562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:35.025562+0000","last_clean_scrub_stamp":"2024-04-29T20:48:35.025562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:56:31.024657+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.695724+0000","last_change":"2024-04-29T20:48:29.696557+0000","last_active":"2024-04-29T20:48:29.695724+0000","last_peered":"2024-04-29T20:48:29.695724+0000","last_clean":"2024-04-29T20:48:29.695724+0000","last_became_active":"2024-04-29T20:22:58.220401+0000","last_became_peered":"2024-04-29T20:22:58.220401+0000","last_unstale":"2024-04-29T20:48:29.695724+0000","last_undegraded":"2024-04-29T20:48:29.695724+0000","last_fullsized":"2024-04-29T20:48:29.695724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.694039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.694039+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.694039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T21:58:02.880900+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1879,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:26.015512+0000","last_change":"2024-04-29T20:48:26.016409+0000","last_active":"2024-04-29T20:48:26.015512+0000","last_peered":"2024-04-29T20:48:26.015512+0000","last_clean":"2024-04-29T20:48:26.015512+0000","last_became_active":"2024-04-29T20:25:05.045385+0000","last_became_peered":"2024-04-29T20:25:05.045385+0000","last_unstale":"2024-04-29T20:48:26.015512+0000","last_undegraded":"2024-04-29T20:48:26.015512+0000","last_fullsized":"2024-04-29T20:48:26.015512+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:26.010545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:26.010545+0000","last_clean_scrub_stamp":"2024-04-29T20:48:26.010545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:06:42.933639+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1885,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.180935+0000","last_change":"2024-04-29T20:48:29.185471+0000","last_active":"2024-04-29T20:48:29.180935+0000","last_peered":"2024-04-29T20:48:29.180935+0000","last_clean":"2024-04-29T20:48:29.180935+0000","last_became_active":"2024-04-29T20:22:59.025610+0000","last_became_peered":"2024-04-29T20:22:59.025610+0000","last_unstale":"2024-04-29T20:48:29.180935+0000","last_undegraded":"2024-04-29T20:48:29.180935+0000","last_fullsized":"2024-04-29T20:48:29.180935+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.176556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.176556+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.176556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:52:06.939200+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1879,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.951744+0000","last_change":"2024-04-29T20:48:28.952605+0000","last_active":"2024-04-29T20:48:28.951744+0000","last_peered":"2024-04-29T20:48:28.951744+0000","last_clean":"2024-04-29T20:48:28.951744+0000","last_became_active":"2024-04-29T20:25:19.206413+0000","last_became_peered":"2024-04-29T20:25:19.206413+0000","last_unstale":"2024-04-29T20:48:28.951744+0000","last_undegraded":"2024-04-29T20:48:28.951744+0000","last_fullsized":"2024-04-29T20:48:28.951744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.949663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.949663+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.949663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:18:18.247413+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1882,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:32.668426+0000","last_change":"2024-04-29T20:48:32.669254+0000","last_active":"2024-04-29T20:48:32.668426+0000","last_peered":"2024-04-29T20:48:32.668426+0000","last_clean":"2024-04-29T20:48:32.668426+0000","last_became_active":"2024-04-29T20:22:59.003236+0000","last_became_peered":"2024-04-29T20:22:59.003236+0000","last_unstale":"2024-04-29T20:48:32.668426+0000","last_undegraded":"2024-04-29T20:48:32.668426+0000","last_fullsized":"2024-04-29T20:48:32.668426+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:32.666794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:32.666794+0000","last_clean_scrub_stamp":"2024-04-29T20:48:32.666794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:31:00.030515+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1868,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.272536+0000","last_change":"2024-04-29T20:48:29.273379+0000","last_active":"2024-04-29T20:48:29.272536+0000","last_peered":"2024-04-29T20:48:29.272536+0000","last_clean":"2024-04-29T20:48:29.272536+0000","last_became_active":"2024-04-29T20:25:19.589991+0000","last_became_peered":"2024-04-29T20:25:19.589991+0000","last_unstale":"2024-04-29T20:48:29.272536+0000","last_undegraded":"2024-04-29T20:48:29.272536+0000","last_fullsized":"2024-04-29T20:48:29.272536+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.270909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.270909+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.270909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T22:57:08.446947+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1861,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:29.943501+0000","last_change":"2024-04-29T20:48:29.944464+0000","last_active":"2024-04-29T20:48:29.943501+0000","last_peered":"2024-04-29T20:48:29.943501+0000","last_clean":"2024-04-29T20:48:29.943501+0000","last_became_active":"2024-04-29T20:25:24.662027+0000","last_became_peered":"2024-04-29T20:25:24.662027+0000","last_unstale":"2024-04-29T20:48:29.943501+0000","last_undegraded":"2024-04-29T20:48:29.943501+0000","last_fullsized":"2024-04-29T20:48:29.943501+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:29.941785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:29.941785+0000","last_clean_scrub_stamp":"2024-04-29T20:48:29.941785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:45:05.066914+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1871,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:35.698561+0000","last_change":"2024-04-29T20:48:35.699401+0000","last_active":"2024-04-29T20:48:35.698561+0000","last_peered":"2024-04-29T20:48:35.698561+0000","last_clean":"2024-04-29T20:48:35.698561+0000","last_became_active":"2024-04-29T20:25:26.576214+0000","last_became_peered":"2024-04-29T20:25:26.576214+0000","last_unstale":"2024-04-29T20:48:35.698561+0000","last_undegraded":"2024-04-29T20:48:35.698561+0000","last_fullsized":"2024-04-29T20:48:35.698561+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:35.696934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:35.696934+0000","last_clean_scrub_stamp":"2024-04-29T20:48:35.696934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T02:41:52.123293+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1882,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:34.059818+0000","last_change":"2024-04-29T20:48:34.060604+0000","last_active":"2024-04-29T20:48:34.059818+0000","last_peered":"2024-04-29T20:48:34.059818+0000","last_clean":"2024-04-29T20:48:34.059818+0000","last_became_active":"2024-04-29T20:25:02.734276+0000","last_became_peered":"2024-04-29T20:25:02.734276+0000","last_unstale":"2024-04-29T20:48:34.059818+0000","last_undegraded":"2024-04-29T20:48:34.059818+0000","last_fullsized":"2024-04-29T20:48:34.059818+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:34.058122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:34.058122+0000","last_clean_scrub_stamp":"2024-04-29T20:48:34.058122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T00:04:27.322259+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1881,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:31.916998+0000","last_change":"2024-04-29T20:48:31.917827+0000","last_active":"2024-04-29T20:48:31.916998+0000","last_peered":"2024-04-29T20:48:31.916998+0000","last_clean":"2024-04-29T20:48:31.916998+0000","last_became_active":"2024-04-29T20:25:06.011110+0000","last_became_peered":"2024-04-29T20:25:06.011110+0000","last_unstale":"2024-04-29T20:48:31.916998+0000","last_undegraded":"2024-04-29T20:48:31.916998+0000","last_fullsized":"2024-04-29T20:48:31.916998+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:31.915289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:31.915289+0000","last_clean_scrub_stamp":"2024-04-29T20:48:31.915289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:24:21.583939+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1883,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:32.989489+0000","last_change":"2024-04-29T20:48:32.990723+0000","last_active":"2024-04-29T20:48:32.989489+0000","last_peered":"2024-04-29T20:48:32.989489+0000","last_clean":"2024-04-29T20:48:32.989489+0000","last_became_active":"2024-04-29T20:25:02.256407+0000","last_became_peered":"2024-04-29T20:25:02.256407+0000","last_unstale":"2024-04-29T20:48:32.989489+0000","last_undegraded":"2024-04-29T20:48:32.989489+0000","last_fullsized":"2024-04-29T20:48:32.989489+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:32.987614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:32.987614+0000","last_clean_scrub_stamp":"2024-04-29T20:48:32.987614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T23:06:29.244537+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:27.295944+0000","last_change":"2024-04-29T20:48:27.297036+0000","last_active":"2024-04-29T20:48:27.295944+0000","last_peered":"2024-04-29T20:48:27.295944+0000","last_clean":"2024-04-29T20:48:27.295944+0000","last_became_active":"2024-04-29T20:25:05.517205+0000","last_became_peered":"2024-04-29T20:25:05.517205+0000","last_unstale":"2024-04-29T20:48:27.295944+0000","last_undegraded":"2024-04-29T20:48:27.295944+0000","last_fullsized":"2024-04-29T20:48:27.295944+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:27.294294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:27.294294+0000","last_clean_scrub_stamp":"2024-04-29T20:48:27.294294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T03:00:01.942619+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1875,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.034568+0000","last_change":"2024-04-29T20:48:28.034568+0000","last_active":"2024-04-29T20:48:28.034568+0000","last_peered":"2024-04-29T20:48:28.034568+0000","last_clean":"2024-04-29T20:48:28.034568+0000","last_became_active":"2024-04-29T20:25:11.174336+0000","last_became_peered":"2024-04-29T20:25:11.174336+0000","last_unstale":"2024-04-29T20:48:28.034568+0000","last_undegraded":"2024-04-29T20:48:28.034568+0000","last_fullsized":"2024-04-29T20:48:28.034568+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.028636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.028636+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.028636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T06:53:55.590130+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1871,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:33.668759+0000","last_change":"2024-04-29T20:48:33.669764+0000","last_active":"2024-04-29T20:48:33.668759+0000","last_peered":"2024-04-29T20:48:33.668759+0000","last_clean":"2024-04-29T20:48:33.668759+0000","last_became_active":"2024-04-29T20:25:09.429696+0000","last_became_peered":"2024-04-29T20:25:09.429696+0000","last_unstale":"2024-04-29T20:48:33.668759+0000","last_undegraded":"2024-04-29T20:48:33.668759+0000","last_fullsized":"2024-04-29T20:48:33.668759+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:33.667101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:33.667101+0000","last_clean_scrub_stamp":"2024-04-29T20:48:33.667101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T07:05:38.466414+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1879,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:33.999917+0000","last_change":"2024-04-29T20:48:34.000740+0000","last_active":"2024-04-29T20:48:33.999917+0000","last_peered":"2024-04-29T20:48:33.999917+0000","last_clean":"2024-04-29T20:48:33.999917+0000","last_became_active":"2024-04-29T20:25:22.551405+0000","last_became_peered":"2024-04-29T20:25:22.551405+0000","last_unstale":"2024-04-29T20:48:33.999917+0000","last_undegraded":"2024-04-29T20:48:33.999917+0000","last_fullsized":"2024-04-29T20:48:33.999917+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:33.996211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:33.996211+0000","last_clean_scrub_stamp":"2024-04-29T20:48:33.996211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:40:28.225676+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1872,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.875567+0000","last_change":"2024-04-29T20:48:28.876595+0000","last_active":"2024-04-29T20:48:28.875567+0000","last_peered":"2024-04-29T20:48:28.875567+0000","last_clean":"2024-04-29T20:48:28.875567+0000","last_became_active":"2024-04-29T20:25:12.263826+0000","last_became_peered":"2024-04-29T20:25:12.263826+0000","last_unstale":"2024-04-29T20:48:28.875567+0000","last_undegraded":"2024-04-29T20:48:28.875567+0000","last_fullsized":"2024-04-29T20:48:28.875567+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.871622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.871622+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.871622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T04:19:26.221756+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1878,"reported_epoch":976,"state":"active+clean","last_fresh":"2024-04-29T20:48:28.030280+0000","last_change":"2024-04-29T20:48:28.031631+0000","last_active":"2024-04-29T20:48:28.030280+0000","last_peered":"2024-04-29T20:48:28.030280+0000","last_clean":"2024-04-29T20:48:28.030280+0000","last_became_active":"2024-04-29T20:25:09.849524+0000","last_became_peered":"2024-04-29T20:25:09.849524+0000","last_unstale":"2024-04-29T20:48:28.030280+0000","last_undegraded":"2024-04-29T20:48:28.030280+0000","last_fullsized":"2024-04-29T20:48:28.030280+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T20:48:28.025453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T20:48:28.025453+0000","last_clean_scrub_stamp":"2024-04-29T20:48:28.025453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T05:14:13.315429+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":22,"seq":94489280827,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81068,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290772,"statfs":{"total":96636764160,"available":96553750528,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280827,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81464,"kb_used_data":5872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290376,"statfs":{"total":96636764160,"available":96553345024,"internally_reserved":0,"allocated":6012928,"data_stored":3244619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280826,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68592,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303248,"statfs":{"total":96636764160,"available":96566525952,"internally_reserved":0,"allocated":5611520,"data_stored":2846779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":20,"seq":85899346237,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81068,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290772,"statfs":{"total":96636764160,"available":96553750528,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81068,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290772,"statfs":{"total":96636764160,"available":96553750528,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509755,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68980,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302860,"statfs":{"total":96636764160,"available":96566128640,"internally_reserved":0,"allocated":6008832,"data_stored":3244614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509756,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72748,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299092,"statfs":{"total":96636764160,"available":96562270208,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509753,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81068,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290772,"statfs":{"total":96636764160,"available":96553750528,"internally_reserved":0,"allocated":5607424,"data_stored":2846774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T20:48:53.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-29T20:48:53.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.321+0000 7f408452d640 10 monclient: get_monmap_and_config 2024-04-29T20:48:53.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.321+0000 7f408452d640 10 monclient: build_initial_monmap 2024-04-29T20:48:53.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.321+0000 7f408452d640 10 monclient: monmap: 2024-04-29T20:48:53.321 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:48:53.322585+0000 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:48:53.322585+0000 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a-legacy 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c-legacy 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-b 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:3: v2:172.21.15.64:3300/0 mon.noname-a 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:4: v2:172.21.15.167:3300/0 mon.noname-c 2024-04-29T20:48:53.322 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.322+0000 7f408452d640 1 Processor -- start 2024-04-29T20:48:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.322+0000 7f408452d640 1 -- start start 2024-04-29T20:48:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.322+0000 7f408452d640 10 monclient: init 2024-04-29T20:48:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-29T20:48:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f407c0b2230 addr v1:172.21.15.64:6789/0 2024-04-29T20:48:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f407c0aa460 addr v1:172.21.15.167:6789/0 2024-04-29T20:48:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f407c0b5c60 0x7f407c14ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient(hunting): picked mon.noname-c con 0x7f407c0b5c60 addr v2:172.21.15.167:3300/0 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407c0ac1f0 con 0x7f407c0b2230 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407c0ac4d0 con 0x7f407c0aa460 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 1 -- --> v2:172.21.15.167:3300/0 -- mon_getmap magic: 0 -- 0x7f407c0aa8d0 con 0x7f407c0b5c60 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f408452d640 10 monclient(hunting): authenticate will time out at 3082.482921s 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.323+0000 7f4082aca640 1 --2- >> v2:172.21.15.167:3300/0 conn(0x7f407c0b5c60 0x7f407c14ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.324+0000 7f4081ac8640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f407c0aa460 0x7f407c0b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.64:48204/0 (socket says 172.21.15.64:48204) 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.324+0000 7f4081ac8640 1 -- 172.21.15.64:0/3504973320 learned_addr learned my addr 172.21.15.64:0/3504973320 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.324+0000 7f4082aca640 10 monclient(hunting): get_auth_request con 0x7f407c0b5c60 auth_method 0 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.324+0000 7f4082aca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.324+0000 7f4082aca640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:53.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.324+0000 7f4082aca640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:53.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.325+0000 7f4082aca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:53.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.325+0000 7f4082aca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:53.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.325+0000 7f4082aca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:53.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 10 monclient(hunting): handle_auth_done global_id 22225 payload 274 2024-04-29T20:48:53.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 1 -- 172.21.15.64:0/3504973320 >> v1:172.21.15.167:6789/0 conn(0x7f407c0aa460 legacy=0x7f407c0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 1 -- 172.21.15.64:0/3504973320 >> v1:172.21.15.64:6789/0 conn(0x7f407c0b2230 legacy=0x7f407c146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 10 monclient: _finish_hunting 0 2024-04-29T20:48:53.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 1 monclient: found mon.noname-c 2024-04-29T20:48:53.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:53.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.167:3300/0 2024-04-29T20:48:53.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.327+0000 7f4082aca640 1 -- 172.21.15.64:0/3504973320 --> v2:172.21.15.167:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407c0b2610 con 0x7f407c0b5c60 2024-04-29T20:48:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.331+0000 7f4082aca640 1 --2- 172.21.15.64:0/3504973320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b5c60 0x7f407c14ecf0 secure :-1 s=READY pgs=5650 cs=0 l=1 rev1=1 crypto rx=0x7f4074009d90 tx=0x7f407402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd82b2d6a958456 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 1 -- 172.21.15.64:0/3504973320 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074039070 con 0x7f407c0b5c60 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: dump: 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: _finish_auth 0 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: _check_auth_tickets 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 1 -- 172.21.15.64:0/3504973320 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407402cd00 con 0x7f407c0b5c60 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f408452d640 5 monclient: authenticate success, global_id 22225 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f408452d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f408452d640 10 monclient: get_monmap_and_config success 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f408452d640 4 set_mon_vals no callback set 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 1 -- 172.21.15.64:0/3504973320 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074034660 con 0x7f407c0b5c60 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.335+0000 7f40812c7640 10 monclient: dump: 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:53.338 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 10 monclient: shutdown 2024-04-29T20:48:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 -- 172.21.15.64:0/3504973320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b5c60 msgr2=0x7f407c14ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 --2- 172.21.15.64:0/3504973320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b5c60 0x7f407c14ecf0 secure :-1 s=READY pgs=5650 cs=0 l=1 rev1=1 crypto rx=0x7f4074009d90 tx=0x7f407402bf50 comp rx=0 tx=0).stop 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 -- 172.21.15.64:0/3504973320 shutdown_connections 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 --2- 172.21.15.64:0/3504973320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b5c60 0x7f407c14ecf0 unknown :-1 s=CLOSED pgs=5650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 -- 172.21.15.64:0/3504973320 >> 172.21.15.64:0/3504973320 conn(0x7f407c06c960 msgr2=0x7f407c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 -- 172.21.15.64:0/3504973320 shutdown_connections 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.336+0000 7f408452d640 1 -- 172.21.15.64:0/3504973320 wait complete. 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 10 monclient: build_initial_monmap 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 10 monclient: monmap: 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-04-29T20:48:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.noname-a 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.noname-b 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.noname-c 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 1 librados: starting msgr at 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 1 librados: starting objecter 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 1 Processor -- start 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 1 -- start start 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 1 librados: setting wanted keys 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 1 librados: calling monclient init 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.337+0000 7f408452d640 10 monclient: init 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient: _reopen_session rank -1 2024-04-29T20:48:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f407c0aa460 0x7f407c143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): picked mon.noname-a con 0x7f407c0aa460 addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b2230 0x7f407c13eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): picked mon.noname-b con 0x7f407c0b2230 addr [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f407c0b5c60 0x7f407c1d86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): picked mon.noname-c con 0x7f407c0b5c60 addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f407c14f3b0 con 0x7f407c0aa460 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f407c14f230 con 0x7f407c0b2230 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): start opening mon connection 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 1 -- --> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] -- mon_getmap magic: 0 -- 0x7f407c14f530 con 0x7f407c0b5c60 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): _renew_subs 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f408452d640 10 monclient(hunting): authenticate will time out at 3082.497793s 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f40822c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f407c0aa460 0x7f407c143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f4082aca640 1 --2- >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f407c0b5c60 0x7f407c1d86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.338+0000 7f4081ac8640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b2230 0x7f407c13eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f407c0aa460 0x7f407c143940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40718/0 (socket says 172.21.15.64:40718) 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 1 -- 172.21.15.64:0/3123210115 learned_addr learned my addr 172.21.15.64:0/3123210115 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): get_auth_request con 0x7f407c0aa460 auth_method 0 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): get_auth_request con 0x7f407c0b2230 auth_method 0 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): get_auth_request con 0x7f407c0b5c60 auth_method 0 2024-04-29T20:48:53.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): _init_auth method 2 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): _init_auth creating new auth 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4081ac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f40822c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.339+0000 7f4082aca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 10 monclient(hunting): handle_auth_done global_id 22231 payload 824 2024-04-29T20:48:53.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 1 -- 172.21.15.64:0/3123210115 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f407c0b5c60 msgr2=0x7f407c1d86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f407c0b5c60 0x7f407c1d86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 1 -- 172.21.15.64:0/3123210115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f407c0aa460 msgr2=0x7f407c143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f407c0aa460 0x7f407c143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 10 monclient: _finish_hunting 0 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 1 monclient: found mon.noname-b 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.167:3300/0 2024-04-29T20:48:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.342+0000 7f4081ac8640 1 -- 172.21.15.64:0/3123210115 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407c0ac9e0 con 0x7f407c0b2230 2024-04-29T20:48:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.344+0000 7f4081ac8640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b2230 0x7f407c13eb40 secure :-1 s=READY pgs=5651 cs=0 l=1 rev1=1 crypto rx=0x7f406c00c470 tx=0x7f406c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.344+0000 7f4072ffd640 10 client.?.objecter ms_handle_connect 0x7f407c0b2230 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.344+0000 7f4072ffd640 10 client.?.objecter resend_mon_ops 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c017070 con 0x7f407c0b2230 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 10 monclient: dump: 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 10 monclient: _finish_auth 0 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 10 monclient: _check_auth_tickets 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c005ce0 con 0x7f407c0b2230 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f408452d640 5 monclient: authenticate success, global_id 22231 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f408452d640 10 monclient: _renew_subs 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f408452d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.347+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407c13f520 con 0x7f407c0b2230 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f4072ffd640 10 monclient: handle_config config(0 keys) 2024-04-29T20:48:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c0072d0 con 0x7f407c0b2230 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f4072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f4072ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f4072ffd640 10 monclient: dump: 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 2216f09b-4679-4a15-bb1e-fe6d8ebb49fc 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-04-29T20:21:08.698911+0000 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f4080ac6640 4 set_mon_vals no callback set 2024-04-29T20:48:53.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 10 client.22231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 10 monclient: _renew_subs 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f407c1d8c30 con 0x7f407c0b2230 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 1 librados: init done 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:53.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.348+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f407c13f310 con 0x7f407c0b2230 2024-04-29T20:48:53.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.369+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f406c01a070 con 0x7f407c0b2230 2024-04-29T20:48:53.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.369+0000 7f4072ffd640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f406006e280 0x7f4060070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-29T20:48:53.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.369+0000 7f40822c9640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f406006e280 0x7f4060070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-29T20:48:53.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.369+0000 7f40822c9640 10 monclient: get_auth_request con 0x7f406006e280 auth_method 0 2024-04-29T20:48:53.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.370+0000 7f40822c9640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f406006e280 0x7f4060070740 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4078001840 tx=0x7f4078009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-29T20:48:53.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.370+0000 7f4072ffd640 10 client.22231.objecter ms_handle_connect 0x7f406006e280 2024-04-29T20:48:53.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.372+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(976..976 src has 638..976) ==== 6337+0+0 (secure 0 0 0) 0x7f406c057150 con 0x7f407c0b2230 2024-04-29T20:48:53.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.372+0000 7f4072ffd640 10 client.22231.objecter ms_dispatch 0x7f407c000b90 osd_map(976..976 src has 638..976) 2024-04-29T20:48:53.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.372+0000 7f4072ffd640 3 client.22231.objecter handle_osd_map got epochs [976,976] > 0 2024-04-29T20:48:53.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.372+0000 7f4072ffd640 3 client.22231.objecter handle_osd_map decoding full epoch 976 2024-04-29T20:48:53.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.372+0000 7f4072ffd640 20 client.22231.objecter dump_active .. 0 homeless 2024-04-29T20:48:53.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.619+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f406c0590b0 con 0x7f407c0b2230 2024-04-29T20:48:53.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.619+0000 7f4072ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-29T20:48:53.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.619+0000 7f4072ffd640 10 monclient: _finish_command 1 = system:0 2024-04-29T20:48:53.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.805+0000 7f408452d640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-29T20:48:53.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.805+0000 7f408452d640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-29T20:48:53.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.805+0000 7f408452d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.167:3300/0 2024-04-29T20:48:53.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.805+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f407c04fce0 con 0x7f407c0b2230 2024-04-29T20:48:53.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f406c05d3b0 con 0x7f407c0b2230 2024-04-29T20:48:53.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4072ffd640 10 monclient: handle_config config(1 keys) 2024-04-29T20:48:53.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4080ac6640 4 set_mon_vals no callback set 2024-04-29T20:48:53.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4080ac6640 10 set_mon_vals mon_health_to_clog = false 2024-04-29T20:48:53.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4072ffd640 1 -- 172.21.15.64:0/3123210115 <== mon.1 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7f406c061be0 con 0x7f407c0b2230 2024-04-29T20:48:53.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4072ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-29T20:48:53.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.961+0000 7f4072ffd640 10 monclient: _finish_command 2 = system:0 2024-04-29T20:48:53.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 10 librados: watch_flush enter 2024-04-29T20:48:53.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 10 librados: watch_flush exit 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 20 client.22231.objecter shutdown clearing up homeless session... 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 10 client.22231.objecter successfully canceled tick 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f406006e280 msgr2=0x7f4060070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f406006e280 0x7f4060070740 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4078001840 tx=0x7f4078009040 comp rx=0 tx=0).stop 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 10 monclient: shutdown 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b2230 msgr2=0x7f407c13eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b2230 0x7f407c13eb40 secure :-1 s=READY pgs=5651 cs=0 l=1 rev1=1 crypto rx=0x7f406c00c470 tx=0x7f406c001e70 comp rx=0 tx=0).stop 2024-04-29T20:48:53.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 shutdown_connections 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:6800/3900323440,v1:172.21.15.167:6801/3900323440] conn(0x7f406006e280 0x7f4060070740 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f407c0b5c60 0x7f407c1d86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f407c0b2230 0x7f407c13eb40 unknown :-1 s=CLOSED pgs=5651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 --2- 172.21.15.64:0/3123210115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f407c0aa460 0x7f407c143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 >> 172.21.15.64:0/3123210115 conn(0x7f407c06c960 msgr2=0x7f407c0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 shutdown_connections 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 -- 172.21.15.64:0/3123210115 wait complete. 2024-04-29T20:48:53.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-29T20:48:53.964+0000 7f408452d640 1 librados: shutdown 2024-04-29T20:48:53.974 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-29T20:48:53.975 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-29T20:48:53.975 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-29T20:48:53.975 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:48:53.988 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:48:53.979+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 32862) UID: 0 2024-04-29T20:48:53.988 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-29T20:48:53.981+0000 ac84640 -1 osd.0 976 *** Got signal Terminated *** 2024-04-29T20:48:57.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-29T20:49:00.076 INFO:tasks.ceph.osd.0:Stopped 2024-04-29T20:49:00.077 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-29T20:49:00.077 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:49:00.090 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:49:00.081+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 32871) UID: 0 2024-04-29T20:49:00.090 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-29T20:49:00.082+0000 ac84640 -1 osd.1 978 *** Got signal Terminated *** 2024-04-29T20:49:03.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-29T20:49:04.781 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:23.289 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.788 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:23.296 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.817 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:23.325 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.836 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:23.315 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.839 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:23.318 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.848 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:23.356 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.849 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:23.357 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.861 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:23.339 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:04.869 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:23.377 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.890 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:27.397 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.892 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:27.400 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.914 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:27.422 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.945 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:27.424 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.947 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:27.425 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.961 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:27.439 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.961 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:27.468 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.963 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:27.470 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:08.985 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:27.492 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.630 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:28.137 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.638 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:28.115 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.681 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:28.159 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.686 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:28.193 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.692 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:28.170 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.697 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:28.205 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.765 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:28.272 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.766 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:28.273 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:09.777 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:28.284 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.704 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:43.211 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.705 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:43.213 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.719 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:43.227 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.756 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:43.234 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.757 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:43.236 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.758 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:43.266 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.760 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:43.267 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.772 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:43.250 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:24.774 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:43.282 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.745 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:48.252 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.748 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:48.256 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.759 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:27:48.266 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.798 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:48.306 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.799 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:48.307 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.801 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:48.280 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.802 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:48.281 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.809 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:27:48.317 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:29.813 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:27:48.291 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.759 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:03.266 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.761 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:03.269 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.775 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:03.283 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.812 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:03.290 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.813 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:03.291 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.814 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:03.322 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.818 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:03.323 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.826 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:03.305 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:44.830 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:03.337 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.773 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:08.251 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.775 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:08.281 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.777 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:08.284 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.777 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:08.255 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.787 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:08.265 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.792 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:08.299 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.832 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:08.339 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.834 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:08.341 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:49.843 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:08.350 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:49:59.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-29T20:50:01.091 INFO:tasks.ceph.osd.1:Stopped 2024-04-29T20:50:01.092 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-29T20:50:01.092 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:50:01.108 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:50:01.096+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 32875) UID: 0 2024-04-29T20:50:01.108 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-29T20:50:01.098+0000 ac84640 -1 osd.2 982 *** Got signal Terminated *** 2024-04-29T20:50:04.940 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:23.447 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:04.942 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:23.449 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:04.963 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:23.471 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:04.992 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:23.470 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:04.993 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:23.471 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:05.000 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:23.507 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:05.001 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:23.509 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:05.011 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:23.489 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:05.029 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:23.537 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:05.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-29T20:50:09.843 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:28.351 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.848 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:28.356 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.859 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:28.366 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.897 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:28.376 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.899 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:28.406 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.899 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:28.378 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.900 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:28.408 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.910 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:28.388 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:09.911 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:28.419 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.886 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:43.393 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.888 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:43.395 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.902 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:43.410 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.940 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:43.418 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.941 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:43.419 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.955 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:43.434 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.957 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:43.465 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.958 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:43.466 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:24.974 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:43.482 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.907 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:48.414 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.910 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:48.388 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.911 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:48.390 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.912 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:48.420 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.925 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:48.403 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.930 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:48.438 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.966 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:48.473 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.968 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:48.476 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:29.983 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:48.491 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:31.599 INFO:tasks.ceph.osd.2:Stopped 2024-04-29T20:50:31.600 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-29T20:50:31.600 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:50:31.613 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:50:31.604+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 32886) UID: 0 2024-04-29T20:50:31.613 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-29T20:50:31.605+0000 ac84640 -1 osd.3 986 *** Got signal Terminated *** 2024-04-29T20:50:39.935 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:58.443 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:39.939 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:58.447 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:39.961 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:28:58.469 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:39.990 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:58.468 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:39.991 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:58.470 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:39.996 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:58.504 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:39.999 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:58.507 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:40.004 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:28:58.483 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:40.019 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:28:58.526 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:54.970 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:13.477 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:54.974 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:13.482 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:54.988 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:13.496 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:55.025 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:13.503 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:55.025 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:13.533 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:55.026 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:13.504 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:55.027 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:13.535 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:55.040 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:13.519 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:55.042 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:13.550 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:59.988 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:18.495 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:59.988 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:18.466 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:59.991 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:18.469 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:50:59.992 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:18.500 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:00.001 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:18.479 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:00.003 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:18.511 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:00.041 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:18.549 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:00.045 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:18.553 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:00.056 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:18.564 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.026 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:33.533 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.027 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:33.535 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.040 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:33.548 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.078 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:33.556 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.078 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:33.557 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.080 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:33.587 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.081 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:33.589 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.093 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:33.572 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:15.096 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:33.603 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.039 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:38.547 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.042 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:38.550 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.053 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:38.561 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.094 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:38.572 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.095 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:38.573 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.095 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:38.602 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.096 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:38.604 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.105 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:38.584 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:20.106 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:38.614 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:30.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-29T20:51:32.615 INFO:tasks.ceph.osd.3:Stopped 2024-04-29T20:51:32.615 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-29T20:51:32.615 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:51:32.635 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:51:32.621+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 32619) UID: 0 2024-04-29T20:51:32.635 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-29T20:51:32.623+0000 ac84640 -1 osd.4 988 *** Got signal Terminated *** 2024-04-29T20:51:35.077 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:53.584 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.079 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:53.587 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.096 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:53.604 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.133 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:53.640 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.134 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:53.642 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.148 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:53.627 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.149 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:53.628 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.150 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:53.658 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:35.166 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:53.644 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:36.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-29T20:51:40.099 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:58.607 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.101 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:58.609 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.111 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:29:58.619 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.147 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:58.626 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.150 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:58.628 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.154 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:58.662 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.156 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:58.664 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.161 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:29:58.640 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:40.166 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:29:58.674 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.084 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:13.592 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.134 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:13.613 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.135 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:13.614 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.140 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:13.648 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.152 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:13.631 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.157 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:13.665 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.194 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:13.702 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.195 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:13.703 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:51:55.226 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:13.734 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.151 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:18.658 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.154 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:18.662 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.164 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:18.672 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.203 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:18.711 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.204 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:18.682 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.204 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:18.713 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.205 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:18.684 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.215 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:18.694 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:00.216 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:18.723 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:09.222 INFO:tasks.ceph.osd.4:Stopped 2024-04-29T20:52:09.223 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-29T20:52:09.223 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:52:09.236 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-29T20:52:09.228+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 32621) UID: 0 2024-04-29T20:52:09.236 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-29T20:52:09.229+0000 ac84640 -1 osd.5 990 *** Got signal Terminated *** 2024-04-29T20:52:15.201 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:33.708 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.203 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:33.711 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.221 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:33.728 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.254 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:33.733 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.255 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:33.734 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.255 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:33.763 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.272 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:33.751 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.278 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:33.786 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.296 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:33.804 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:15.326 INFO:tasks.ceph.osd.5:Stopped 2024-04-29T20:52:15.326 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-29T20:52:15.326 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:52:15.344 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:52:15.331+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 32625) UID: 0 2024-04-29T20:52:15.345 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-29T20:52:15.333+0000 ac84640 -1 osd.6 992 *** Got signal Terminated *** 2024-04-29T20:52:20.216 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:38.723 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.220 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:38.728 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.235 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:38.743 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.268 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:38.747 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.270 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:38.778 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.272 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:38.751 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.274 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:38.782 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.284 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:38.763 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:20.285 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:38.793 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.268 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:53.776 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.271 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:53.779 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.286 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:53.794 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.319 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:53.798 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.320 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:53.799 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.335 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:53.813 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.370 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:53.878 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.372 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:53.880 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:35.390 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:53.898 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.272 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:58.779 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.275 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:58.784 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.287 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:30:58.795 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.323 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:58.802 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.324 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:58.803 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.327 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:58.835 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.329 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:58.837 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.335 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:30:58.814 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:40.340 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:30:58.848 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.475 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:04.983 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.529 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:05.037 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.545 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:05.053 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.582 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:05.061 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.583 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:05.062 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.587 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:05.095 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.591 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:05.098 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.598 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:05.077 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:46.607 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:05.115 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.282 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:13.790 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.336 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:13.843 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.347 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:13.855 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.384 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:13.863 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.385 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:13.864 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.396 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:13.875 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.400 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:13.908 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.405 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:13.913 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:52:55.416 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:13.924 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.345 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:18.853 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.350 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:18.858 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.361 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:18.869 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.399 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:18.877 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.399 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:18.879 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.401 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:18.909 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.403 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:18.911 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.410 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:18.889 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:00.414 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:18.922 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.391 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:33.899 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.394 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:33.902 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.409 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:33.917 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.443 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:33.921 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.444 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:33.923 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.451 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:33.959 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.454 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:33.962 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.467 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:33.946 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:15.468 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:33.976 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.407 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:38.915 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.411 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:38.919 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.427 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:38.935 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.456 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:38.935 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.458 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:38.937 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.469 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:38.947 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.469 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:38.977 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.470 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:38.978 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:20.482 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:38.990 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.499 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:54.007 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.531 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:54.039 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.551 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:54.030 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.553 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:54.061 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.556 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:54.063 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.579 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:54.057 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.585 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:54.094 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.602 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:54.110 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:35.604 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:54.083 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.483 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:58.990 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.485 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:58.994 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.502 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:31:59.010 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.535 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:59.014 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.536 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:59.016 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.558 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:31:59.037 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.561 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:59.069 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.562 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:59.070 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:40.581 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:31:59.089 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.528 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:14.035 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.530 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:14.038 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.545 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:14.053 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.583 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:14.062 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.584 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:14.092 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.584 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:14.064 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.585 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:14.093 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.600 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:14.108 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:55.600 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:14.079 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:53:59.051 INFO:tasks.ceph.osd.6:Stopped 2024-04-29T20:53:59.052 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-29T20:53:59.052 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:53:59.066 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-29T20:53:59.057+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 32626) UID: 0 2024-04-29T20:53:59.069 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-29T20:53:59.059+0000 ac84640 -1 osd.7 992 *** Got signal Terminated *** 2024-04-29T20:54:00.544 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:19.052 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.546 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:19.054 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.561 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:19.069 32571== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.593 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:19.071 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.596 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:19.075 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.600 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:19.109 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.603 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:19.111 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.608 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:19.087 32010== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:00.618 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:19.124 32577== Warning: unimplemented fcntl command: 1036 2024-04-29T20:54:05.153 INFO:tasks.ceph.osd.7:Stopped 2024-04-29T20:54:05.154 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-29T20:54:05.154 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-29T20:54:05.154 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:54:05.178 INFO:tasks.ceph.mgr.y:Stopped 2024-04-29T20:54:05.179 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-29T20:54:05.179 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:54:05.180 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:54:05.180+0000 7f33f9122640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 32008) UID: 0 2024-04-29T20:54:05.180 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-29T20:54:05.180+0000 7f33f9122640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-29T20:54:05.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-29T20:54:05.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-04-29T20:54:05.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-04-29T20:54:05.208 INFO:tasks.ceph.mgr.x:Stopped 2024-04-29T20:54:05.208 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-29T20:54:05.208 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-29T20:54:05.208 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:54:05.234 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-29T20:54:05.224+0000 9fa6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 32547) UID: 0 2024-04-29T20:54:05.235 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-29T20:54:05.226+0000 9fa6640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-29T20:54:05.821 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:24.329 32571== 2024-04-29T20:54:05.821 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:32:24.329 32571== Exit program on first error (--exit-on-first-error=yes) 2024-04-29T20:54:05.841 INFO:tasks.ceph.mon.a.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-04-29T20:54:10.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-04-29T20:54:10.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-29T20:54:10.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-04-29T20:54:10.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-04-29T20:54:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-04-29T20:54:11.311 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-04-29T20:54:11.312 INFO:tasks.ceph.mon.a:Stopped 2024-04-29T20:54:11.313 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-29T20:54:11.313 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:54:11.330 INFO:tasks.ceph.mon.c.smithi064.stderr:2024-04-29T20:54:11.318+0000 9fa6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 32549) UID: 0 2024-04-29T20:54:11.333 INFO:tasks.ceph.mon.c.smithi064.stderr:2024-04-29T20:54:11.320+0000 9fa6640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-04-29T20:54:11.912 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:30.420 32577== 2024-04-29T20:54:11.912 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:32:30.420 32577== Exit program on first error (--exit-on-first-error=yes) 2024-04-29T20:54:11.929 INFO:tasks.ceph.mon.c.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-04-29T20:54:15.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-29T20:54:15.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-04-29T20:54:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-04-29T20:54:17.415 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-04-29T20:54:17.415 INFO:tasks.ceph.mon.c:Stopped 2024-04-29T20:54:17.415 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-29T20:54:17.415 INFO:teuthology.orchestra.run:waiting for 300 2024-04-29T20:54:17.431 INFO:tasks.ceph.mon.b.smithi167.stderr:2024-04-29T20:54:17.421+0000 9fa6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 32005) UID: 0 2024-04-29T20:54:17.432 INFO:tasks.ceph.mon.b.smithi167.stderr:2024-04-29T20:54:17.422+0000 9fa6640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-04-29T20:54:18.120 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:36.599 32010== 2024-04-29T20:54:18.120 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:32:36.599 32010== Exit program on first error (--exit-on-first-error=yes) 2024-04-29T20:54:18.137 INFO:tasks.ceph.mon.b.smithi167.stderr:daemon-helper: command failed with exit status 42 2024-04-29T20:54:20.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-04-29T20:54:20.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-29T20:54:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-04-29T20:54:23.517 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-04-29T20:54:23.517 INFO:tasks.ceph.mon.b:Stopped 2024-04-29T20:54:23.517 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-29T20:54:23.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-04-29T20:54:23.554 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:54:23.555 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-29T20:54:23.682 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:54:23.682 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-29T20:54:23.763 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:54:23.763 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-29T20:54:23.821 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:54:23.821 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-29T20:54:23.880 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:54:23.881 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-29T20:54:24.004 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:54:24.005 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-29T20:54:24.059 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:54:24.059 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-29T20:54:24.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:54:24.115 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-29T20:54:24.182 INFO:tasks.ceph:Archiving mon data... 2024-04-29T20:54:24.186 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/data/mon.a.tgz 2024-04-29T20:54:24.186 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-04-29T20:54:24.203 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.trbdH7OUCl 2024-04-29T20:54:24.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.trbdH7OUCl 2024-04-29T20:54:24.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.trbdH7OUCl 2024-04-29T20:54:24.467 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.trbdH7OUCl is 546KB 2024-04-29T20:54:24.568 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.trbdH7OUCl 2024-04-29T20:54:24.583 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/data/mon.c.tgz 2024-04-29T20:54:24.583 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-04-29T20:54:24.638 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.TxJZBapQKt 2024-04-29T20:54:24.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.TxJZBapQKt 2024-04-29T20:54:24.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.TxJZBapQKt 2024-04-29T20:54:24.885 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.TxJZBapQKt is 546KB 2024-04-29T20:54:24.956 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.TxJZBapQKt 2024-04-29T20:54:24.971 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/data/mon.b.tgz 2024-04-29T20:54:24.971 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-04-29T20:54:24.989 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.B5XtCw0Rfw 2024-04-29T20:54:24.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.B5XtCw0Rfw 2024-04-29T20:54:25.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.B5XtCw0Rfw 2024-04-29T20:54:25.254 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.B5XtCw0Rfw is 546KB 2024-04-29T20:54:25.353 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.B5XtCw0Rfw 2024-04-29T20:54:25.367 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-29T20:54:25.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-29T20:54:25.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-29T20:54:25.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-04-29T20:54:25.696 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-29T20:54:25.962 INFO:tasks.ceph:Archiving crash dumps... 2024-04-29T20:54:25.964 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/remote/smithi064/crash 2024-04-29T20:54:25.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-29T20:54:25.990 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/remote/smithi167/crash 2024-04-29T20:54:25.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-29T20:54:26.014 INFO:tasks.ceph:Compressing logs... 2024-04-29T20:54:26.014 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-29T20:54:26.034 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-29T20:54:26.057 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-04-29T20:54:26.057 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37260.log 2024-04-29T20:54:26.057 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-04-29T20:54:26.057 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.37260.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.105543.log 2024-04-29T20:54:26.057 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37260.log.gz 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105964.log 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37665.log 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105964.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105543.log.gz 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36473.log 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37665.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26767.log 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105964.log.gz/var/log/ceph/ceph-client.admin.36473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36958.log 2024-04-29T20:54:26.074 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37665.log.gz 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.26767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26767.log.gz 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36473.log.gz 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106174.log 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36958.log: /var/log/ceph/ceph-client.admin.40970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105586.log 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36958.log.gz 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-04-29T20:54:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32589.log 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106174.log.gz 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38170.log: /var/log/ceph/ceph-client.admin.32589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37832.log 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105586.log.gz 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36782.log 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38837.log 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.32589.log.gz 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37187.log 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37832.log.gz 2024-04-29T20:54:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38837.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36782.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105880.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37187.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36962.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106090.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105880.log: /var/log/ceph/ceph-client.admin.36962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3%/var/log/ceph/ceph-client.admin.106090.log: -- replaced with /var/log/ceph/ceph-client.admin.105880.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40409.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36962.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38837.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48265.log 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106090.log.gz 2024-04-29T20:54:26.077 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38953.log: gzip/var/log/ceph/ceph-client.admin.40409.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39024.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37328.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39024.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40409.log.gz 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48265.log.gz 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36616.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37328.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37698.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr: 87.5%/var/log/ceph/ceph-client.admin.36616.log: -- replaced with /var/log/ceph/ceph-client.admin.37328.log.gz 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:gzip 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39024.log.gz 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37330.log 2024-04-29T20:54:26.078 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37698.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36616.log.gz 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37239.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37330.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.39220.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37698.log.gz 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37239.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37330.log.gz 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr: -- /var/log/ceph/ceph-osd.3.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39220.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37239.log.gz 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36955.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39012.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36955.log.gz 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-04-29T20:54:26.079 INFO:teuthology.orchestra.run.smithi064.stderr: 92.8%/var/log/ceph/ceph-client.admin.39103.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-29T20:54:26.080 INFO:teuthology.orchestra.run.smithi064.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-04-29T20:54:26.100 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38942.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39276.log: 90.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37496.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38780.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38942.log: /var/log/ceph/ceph-client.admin.37496.log: 87.3% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37496.log.gz 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38942.log.gz 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106006.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36963.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106006.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106006.log.gz 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36366.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36963.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36963.log.gz 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38320.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36366.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36366.log.gz 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2024-04-29T20:54:26.101 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38320.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38780.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39190.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38513.log 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38513.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38757.log 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40641.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38757.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-04-29T20:54:26.102 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105796.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38513.log.gz 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32255.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.32255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32174.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log/var/log/ceph/ceph.log: 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32255.log.gz 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.32174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32455.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32335.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32174.log.gz 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.32455.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.32335.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.32455.log.gz 2024-04-29T20:54:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32215.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32335.log.gz 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.32215.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32215.log.gz 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/mon.b.log: 69.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr: -- /var/log/ceph/valgrind/osd.7.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose/var/log/ceph/valgrind/osd.4.log: -- /var/log/ceph/valgrind/osd.5.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/osd.5.log: 64.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-04-29T20:54:26.104 INFO:teuthology.orchestra.run.smithi167.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32295.log 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32415.log 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.32295.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32375.log 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr: 87.9%/var/log/ceph/ceph-client.admin.32415.log: -- replaced with /var/log/ceph/ceph-client.admin.32295.log.gz 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32415.log.gz/var/log/ceph/ceph-client.admin.32375.log: 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32375.log.gz 2024-04-29T20:54:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-29T20:54:26.108 INFO:teuthology.orchestra.run.smithi064.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.38757.log.gz 2024-04-29T20:54:26.108 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106426.log 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40969.log 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106426.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105796.log.gz 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105838.log 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106426.log.gz 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38184.log 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105838.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105838.log.gz 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38258.log 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38184.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.40969.log.gz 2024-04-29T20:54:26.109 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38637.log 2024-04-29T20:54:26.112 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38258.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-29T20:54:26.121 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38849.log 2024-04-29T20:54:26.128 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38637.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.38637.log.gz 2024-04-29T20:54:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106386.log 2024-04-29T20:54:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38849.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38849.log.gz 2024-04-29T20:54:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39334.log 2024-04-29T20:54:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106386.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106386.log.gz 2024-04-29T20:54:26.159 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105628.log 2024-04-29T20:54:26.170 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39334.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.38184.log.gz 2024-04-29T20:54:26.171 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-29T20:54:26.172 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105628.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105628.log.gz 2024-04-29T20:54:26.172 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48261.log 2024-04-29T20:54:26.174 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.38258.log.gz 2024-04-29T20:54:26.175 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-04-29T20:54:26.175 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48261.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39334.log.gz 2024-04-29T20:54:26.177 INFO:teuthology.orchestra.run.smithi064.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.48261.log.gz 2024-04-29T20:54:26.178 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-04-29T20:54:26.179 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-04-29T20:54:26.179 INFO:teuthology.orchestra.run.smithi064.stderr: 64.0%/var/log/ceph/valgrind/mon.a.log: -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-04-29T20:54:26.179 INFO:teuthology.orchestra.run.smithi064.stderr: 69.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-04-29T20:54:26.179 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-04-29T20:54:26.179 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.3.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-04-29T20:54:26.181 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-04-29T20:54:26.181 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-04-29T20:54:26.181 INFO:teuthology.orchestra.run.smithi064.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz/var/log/ceph/valgrind/mon.c.log: 2024-04-29T20:54:26.181 INFO:teuthology.orchestra.run.smithi064.stderr: 69.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-04-29T20:54:26.181 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106048.log 2024-04-29T20:54:26.182 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.2.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-04-29T20:54:26.182 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-04-29T20:54:26.182 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106048.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-29T20:54:26.183 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38509.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106048.log.gz 2024-04-29T20:54:26.183 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106258.log 2024-04-29T20:54:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gz 2024-04-29T20:54:26.190 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-04-29T20:54:26.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106258.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106258.log.gz 2024-04-29T20:54:26.191 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36960.log 2024-04-29T20:54:26.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39406.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-04-29T20:54:26.195 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-04-29T20:54:26.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36960.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36960.log.gz 2024-04-29T20:54:26.198 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38751.log 2024-04-29T20:54:26.198 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37194.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-04-29T20:54:26.198 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36961.log 2024-04-29T20:54:26.223 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40973.log 2024-04-29T20:54:26.224 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36961.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36961.log.gz 2024-04-29T20:54:26.225 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37874.log 2024-04-29T20:54:26.229 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40973.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40973.log.gz 2024-04-29T20:54:26.233 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38654.log 2024-04-29T20:54:26.247 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37874.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37874.log.gz 2024-04-29T20:54:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105754.log 2024-04-29T20:54:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38654.log: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37192.log 2024-04-29T20:54:26.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105754.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105754.log.gz 2024-04-29T20:54:26.267 INFO:teuthology.orchestra.run.smithi064.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.38751.log.gz 2024-04-29T20:54:26.267 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38654.log.gz 2024-04-29T20:54:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38220.log 2024-04-29T20:54:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37192.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37192.log.gz 2024-04-29T20:54:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106300.log 2024-04-29T20:54:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38220.log: 93.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37616.log 2024-04-29T20:54:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106300.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106300.log.gz 2024-04-29T20:54:26.288 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38220.log.gz 2024-04-29T20:54:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-04-29T20:54:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37616.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37616.log.gz 2024-04-29T20:54:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32891.log 2024-04-29T20:54:26.295 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38267.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-04-29T20:54:26.295 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36964.log 2024-04-29T20:54:26.296 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32891.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.32891.log.gz 2024-04-29T20:54:26.297 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40473.log 2024-04-29T20:54:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36964.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36964.log.gz 2024-04-29T20:54:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-04-29T20:54:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40473.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40473.log.gz 2024-04-29T20:54:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36698.log 2024-04-29T20:54:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38288.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-04-29T20:54:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38538.log 2024-04-29T20:54:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36698.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36698.log.gz 2024-04-29T20:54:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39395.log 2024-04-29T20:54:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38538.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38538.log.gz 2024-04-29T20:54:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38464.log 2024-04-29T20:54:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37576.log 2024-04-29T20:54:26.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-04-29T20:54:26.342 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105712.log 2024-04-29T20:54:26.342 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39201.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37576.log.gz 2024-04-29T20:54:26.350 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38237.log 2024-04-29T20:54:26.351 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105712.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105712.log.gz 2024-04-29T20:54:26.354 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38464.log.gz 2024-04-29T20:54:26.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36035.log 2024-04-29T20:54:26.377 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38237.log: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36740.log 2024-04-29T20:54:26.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36035.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38237.log.gz 2024-04-29T20:54:26.383 INFO:teuthology.orchestra.run.smithi064.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.36035.log.gz 2024-04-29T20:54:26.384 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39395.log.gz 2024-04-29T20:54:26.388 INFO:teuthology.orchestra.run.smithi064.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-04-29T20:54:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-04-29T20:54:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36740.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36740.log.gz 2024-04-29T20:54:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37790.log 2024-04-29T20:54:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39083.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-04-29T20:54:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32918.log 2024-04-29T20:54:26.398 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37790.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37790.log.gz 2024-04-29T20:54:26.398 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48266.log 2024-04-29T20:54:26.399 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32918.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32918.log.gz 2024-04-29T20:54:26.399 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37516.log 2024-04-29T20:54:26.402 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48266.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.48266.log.gz 2024-04-29T20:54:26.403 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39353.log 2024-04-29T20:54:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37516.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37516.log.gz 2024-04-29T20:54:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2024-04-29T20:54:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39353.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.39353.log.gz 2024-04-29T20:54:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33804.log 2024-04-29T20:54:26.420 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40707.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2024-04-29T20:54:26.420 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106344.log 2024-04-29T20:54:26.420 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33804.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33804.log.gz 2024-04-29T20:54:26.422 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-04-29T20:54:26.422 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106344.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106344.log.gz 2024-04-29T20:54:26.423 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106216.log 2024-04-29T20:54:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38928.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-04-29T20:54:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39026.log 2024-04-29T20:54:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106216.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106216.log.gz 2024-04-29T20:54:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105922.log 2024-04-29T20:54:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39026.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.39026.log.gz 2024-04-29T20:54:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106132.log 2024-04-29T20:54:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105922.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105922.log.gz 2024-04-29T20:54:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40474.log 2024-04-29T20:54:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.106132.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106132.log.gz 2024-04-29T20:54:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36656.log 2024-04-29T20:54:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40474.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.40474.log.gz 2024-04-29T20:54:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37681.log 2024-04-29T20:54:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36656.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36656.log.gz 2024-04-29T20:54:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-04-29T20:54:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37681.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37681.log.gz 2024-04-29T20:54:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-04-29T20:54:26.443 INFO:teuthology.orchestra.run.smithi167.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-29T20:54:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40410.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-04-29T20:54:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38633.log 2024-04-29T20:54:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40642.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-04-29T20:54:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-04-29T20:54:26.451 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38633.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.38633.log.gz 2024-04-29T20:54:26.451 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37257.log 2024-04-29T20:54:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33032.log 2024-04-29T20:54:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37257.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-04-29T20:54:26.478 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34732.log 2024-04-29T20:54:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33032.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33032.log.gz 2024-04-29T20:54:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36959.log 2024-04-29T20:54:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34732.log: 87.5% 92.1% -- replaced with /var/log/ceph/ceph-client.admin.34732.log.gz 2024-04-29T20:54:26.487 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-04-29T20:54:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-04-29T20:54:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36959.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36959.log.gz 2024-04-29T20:54:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38954.log 2024-04-29T20:54:26.503 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-04-29T20:54:26.504 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38954.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38954.log.gz 2024-04-29T20:54:26.518 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105670.log 2024-04-29T20:54:26.520 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40472.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-04-29T20:54:26.533 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-04-29T20:54:26.534 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105670.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105670.log.gz 2024-04-29T20:54:26.540 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40971.log 2024-04-29T20:54:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40972.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-04-29T20:54:26.549 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-04-29T20:54:26.552 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40971.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.40971.log.gz 2024-04-29T20:54:26.566 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2024-04-29T20:54:26.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40705.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-04-29T20:54:26.577 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38204.log 2024-04-29T20:54:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38859.log: 94.3% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-04-29T20:54:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38929.log 2024-04-29T20:54:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38204.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2024-04-29T20:54:26.597 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38204.log.gz 2024-04-29T20:54:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38434.log 2024-04-29T20:54:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38929.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38929.log.gz 2024-04-29T20:54:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36560.log 2024-04-29T20:54:26.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105491.log 2024-04-29T20:54:26.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36560.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36560.log.gz 2024-04-29T20:54:26.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.105491.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105491.log.gz 2024-04-29T20:54:27.142 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.38434.log.gz 2024-04-29T20:54:27.387 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38320.log.gz 2024-04-29T20:54:28.280 INFO:teuthology.orchestra.run.smithi167.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-29T20:54:28.383 INFO:teuthology.orchestra.run.smithi064.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-29T20:54:30.108 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-29T20:54:30.673 INFO:teuthology.orchestra.run.smithi167.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-29T20:54:30.746 INFO:teuthology.orchestra.run.smithi167.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-29T20:54:31.412 INFO:teuthology.orchestra.run.smithi167.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-29T20:54:31.493 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-29T20:54:31.580 INFO:teuthology.orchestra.run.smithi167.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-29T20:54:31.581 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-29T20:54:31.581 INFO:teuthology.orchestra.run.smithi167.stderr:real 0m5.512s 2024-04-29T20:54:31.581 INFO:teuthology.orchestra.run.smithi167.stderr:user 0m21.816s 2024-04-29T20:54:31.582 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m0.714s 2024-04-29T20:54:31.602 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-29T20:54:32.201 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-29T20:54:32.262 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-29T20:54:32.264 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-29T20:54:32.264 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m6.217s 2024-04-29T20:54:32.264 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m30.979s 2024-04-29T20:54:32.264 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.998s 2024-04-29T20:54:32.264 INFO:tasks.ceph:Archiving logs... 2024-04-29T20:54:32.264 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/remote/smithi064/log 2024-04-29T20:54:32.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-29T20:54:36.439 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/remote/smithi167/log 2024-04-29T20:54:36.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-29T20:54:39.692 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-29T20:54:39.693 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-29T20:54:39.702 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_354447ca5357e926795d009be846687f04556beb/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-29T20:54:39.703 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-29T20:54:39.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-29T20:54:39.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-29T20:54:39.800 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-29T20:54:39.809 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-29T20:54:39.809 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T20:54:39.812 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T20:54:39.826 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-04-29T20:54:39.827 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-04-29T20:54:39.830 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T20:54:39.830 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-04-29T20:54:39.830 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 51 +1393us[+1409us] +/- 44ms 2024-04-29T20:54:39.831 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 -1150us[-1134us] +/- 71ms 2024-04-29T20:54:39.831 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -1039us[-1039us] +/- 66ms 2024-04-29T20:54:39.831 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:54:39.832 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T20:54:39.832 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-04-29T20:54:39.832 INFO:teuthology.orchestra.run.smithi167.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +1413us[+1442us] +/- 44ms 2024-04-29T20:54:39.832 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 48 -1124us[-1124us] +/- 71ms 2024-04-29T20:54:39.832 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -1028us[-1028us] +/- 66ms 2024-04-29T20:54:39.832 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T20:54:39.833 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-29T20:54:39.841 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-29T20:54:39.842 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-29T20:54:39.850 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-29T20:54:39.939 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-29T20:54:40.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-29T20:54:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:54:40.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-29T20:54:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:54:40.060 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-29T20:54:40.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-29T20:54:40.077 INFO:teuthology.task.internal:Duration was 2986.865765 seconds 2024-04-29T20:54:40.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-29T20:54:40.086 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-29T20:54:40.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-29T20:54:40.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-29T20:54:40.126 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T20:54:40.138 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T20:54:40.456 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-29T20:54:40.456 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-04-29T20:54:40.457 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-29T20:54:40.511 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-04-29T20:54:40.512 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-29T20:54:40.541 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-29T20:54:40.541 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-29T20:54:40.555 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-29T20:54:40.607 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-29T20:54:40.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-29T20:54:40.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-29T20:54:40.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-29T20:54:40.754 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-29T20:54:40.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-29T20:54:40.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-29T20:54:40.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-29T20:54:40.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-29T20:54:40.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-29T20:54:40.882 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-04-29T20:54:40.883 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-04-29T20:54:40.909 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-29T20:54:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:54:40.949 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-29T20:54:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T20:54:40.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-29T20:54:40.974 INFO:teuthology.task.internal:Transferring archived files... 2024-04-29T20:54:40.975 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/remote/smithi064 2024-04-29T20:54:40.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-29T20:54:41.047 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692/remote/smithi167 2024-04-29T20:54:41.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-29T20:54:41.079 INFO:teuthology.task.internal:Removing archive directory... 2024-04-29T20:54:41.079 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-29T20:54:41.082 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-29T20:54:41.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-29T20:54:41.144 INFO:teuthology.task.internal:Not uploading archives. 2024-04-29T20:54:41.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-29T20:54:41.154 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-29T20:54:41.154 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-29T20:54:41.157 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-29T20:54:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 29 20:54 /home/ubuntu/cephtest 2024-04-29T20:54:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 29 20:54 /home/ubuntu/cephtest 2024-04-29T20:54:41.184 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-29T20:54:41.194 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-29T20:54:41.204 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-04-29T20:54:41.205 INFO:teuthology.nuke:Checking targets against current locks 2024-04-29T20:54:41.232 DEBUG:teuthology.nuke:shortname: smithi064 2024-04-29T20:54:41.232 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T20:54:41.258 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 19:58:28.948395', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNd2mtGOrCmynAlbVYW38ivB5bNl5QYQUD0ObRY1s05WIO5wjEpeyExsQSztEFoky8ntBJ1vZjctQHDJnc1VKaE='} 2024-04-29T20:54:41.262 DEBUG:teuthology.nuke:shortname: smithi167 2024-04-29T20:54:41.262 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T20:54:41.283 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-04-29_19:49:34-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7678692', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 19:58:28.950488', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKi49WANi6PnJX4+APe3vVQt93GlNunHK+tM60aoulD8adVf1+KW+zXdQtOTHwjo77GVRci9+1MwUWCFqAxtVwk='} 2024-04-29T20:54:41.315 INFO:teuthology.orchestra.console.smithi064:Power off 2024-04-29T20:54:41.315 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-29T20:54:41.331 INFO:teuthology.orchestra.console.smithi167:Power off 2024-04-29T20:54:41.331 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-29T20:54:41.342 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-04-29T20:54:41.342 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:41.354 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-04-29T20:54:41.354 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:41.364 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-29T20:54:41.376 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-29T20:54:45.365 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:45.378 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:45.475 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-29T20:54:45.487 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-04-29T20:54:49.478 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:49.487 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:49.588 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-29T20:54:49.602 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-04-29T20:54:49.702 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-04-29T20:54:53.590 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T20:54:53.700 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-04-29T20:54:53.801 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-04-29T20:54:53.905 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 2986.8657648563385 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_lflores@teuthology status: fail success: false 2024-04-29T20:54:53.906 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-29T20:54:53.976 INFO:teuthology.run:FAIL